Pipelines & scenarios · Core Pipeline Concepts

What is a backfill?

Mediumbackfillreprocessingpartitionsopsidempotency

Interview question

What is a data backfill, and how do you run one safely?

How to answer

The short answer

A reprocesses historical data after a bug fix, new column, or logic change so curated tables catch up for past dates.

Output
Bug in tax logic from Sept 1
Fix merges on Sept 10

Backfill: re-run transforms for Sept 1 .. Sept 9
(parameterized by run_date, idempotent writes)

Safe backfill habits

  1. 1Parameterize the job (run_date / interval).
  2. 2Ensure writes are idempotent (replace or merge).
  3. 3Bound the range; do not full-scan 5 years on day one.
  4. 4Throttle parallelism to protect the warehouse/cluster cost.
  5. 5Validate row counts / checksums before flipping BI.

Backfill vs catchup

  • Backfill: intentional historical replay you control.
  • : orchestrator automatically filling missed scheduled intervals.

Streaming note

Replay from Kafka retention, raw lake files, or source APIs. If retention is short, the raw lake is your system of replay.

Interview tip: Emphasize parameterized, idempotent, cost-aware replay, not "just rerun prod for all time."