The short answer
A safely retryable pipeline can be re-run after failure without duplicating or corrupting outputs. is the core design goal.
Pattern: stage → validate → atomic publish
1) write to temp / staging path or staging table
2) run DQ checks
3) swap / MERGE overwrite target partition
(or MERGE on keys)Design rules
- 1 or key your writes (
dt=...or business ). - 2Prefer overwrite/merge over blind append.
- 3Commit offsets / watermarks only after success.
- 4Make side effects safe (emails, webhooks: dedupe tokens).
- 5Keep raw inputs durable so you can replay.
Bad vs good
BAD: INSERT into fct; commit Kafka offset; crash; retry inserts again GOOD: write staging → merge fct → then commit offset
Orchestrator settings
Retries with backoff for transient errors; fail and alert for poison data after N tries; quarantine bad files instead of blocking forever.
Interview tip: Say "retryable = idempotent outputs + late watermark/offset commit + atomic publish." That sentence is the answer.