The short answer
A reprocesses historical data after a bug fix, new column, or logic change so curated tables catch up for past dates.
Bug in tax logic from Sept 1
Fix merges on Sept 10
Backfill: re-run transforms for Sept 1 .. Sept 9
(parameterized by run_date, idempotent writes)Safe backfill habits
- 1Parameterize the job (
run_date/ interval). - 2Ensure writes are idempotent (replace or merge).
- 3Bound the range; do not full-scan 5 years on day one.
- 4Throttle parallelism to protect the warehouse/cluster cost.
- 5Validate row counts / checksums before flipping BI.
Backfill vs catchup
- Backfill: intentional historical replay you control.
- : orchestrator automatically filling missed scheduled intervals.
Streaming note
Replay from Kafka retention, raw lake files, or source APIs. If retention is short, the raw lake is your system of replay.
Interview tip: Emphasize parameterized, idempotent, cost-aware replay, not "just rerun prod for all time."