Pipelines & scenarios · Design & Scenarios

Pipeline fails halfway

Hardfailureincidentreplayidempotencyopsscenario

Interview question

What do you do when a pipeline fails halfway through?

How to answer

The short answer

Stay calm, stop the bleed, understand what committed, then repair with an idempotent replay.

Step-by-step response

  1. 1Check orchestrator state: which task failed, error, attempt count.
  2. 2Assess partial writes: is the target incomplete or uncommitted?
  3. 3Do not mark success or silently continue downstream.
  4. 4Fix root cause (bad file, OOM, permissions, poison row).
  5. 5Replay safely: overwrite the partition / merge from last checkpoint.
  6. 6Validate: row counts, null checks, compare to source.
  7. 7Communicate: freshness impact to BI owners if is at risk.

Design that makes this easy

Stage-then-publish, partitioned overwrites, durable raw zone, clear runbooks.

Interview tip: Interviewers want process + idempotent recovery, not "I would rerun the " alone. Say what you check before hitting retry.