Pipelines & scenarios · Design & Scenarios

Handle a source schema change

Hardschema evolutioncontractsdriftbackfillcompatibility

Interview question

How do you handle a source schema change in a data pipeline?

How to answer

The short answer

Sources add, rename, or type-change columns. Pipelines must detect drift, avoid silent corruption, and evolve curated models safely.

Output
Producer adds orders.discount_code
        |
Ingestion lands new field in bronze (good: keep raw flexible)
        |
Contract / schema check
  - alert if unexpected break (type change, dropped column)
  - allow additive fields if policy says so
        |
Silver/gold models updated
  - dbt model + tests
  - backfill if historical values needed

Practical playbook

  1. 1Land raw as-is ( or wide bronze) so you do not lose new fields.
  2. 2Detect drift: schema registry, Great Expectations, dbt tests, contract CI.
  3. 3Classify the change: additive (usually safe) vs breaking (rename/drop/type).
  4. 4Version contracts with producers when possible.
  5. 5Evolve silver/gold explicitly; do not auto-remap renames blindly.
  6. 6 if BI needs history for the new column.

Breaking change example

amount string → number. Gate the deploy, fix casts, validate samples, then promote.

Interview tip: Separate "absorb in raw" from "consciously evolve curated." Mention contracts and alerts so schema changes are not discovered by broken dashboards.