Data quality · Contracts & Ops

Circuit breaker for data pipelines

Hardcircuit breakerquality gatefail fastpublish

Interview question

What is a circuit breaker pattern in data pipelines / data quality?

How to answer

The short answer

A stops downstream work when upstream data looks dangerously bad, so you do not publish poison into marts and dashboards.

Inspired by resilience patterns in services: after repeated failures (or a severe quality breach), open the circuit, fail fast, and wait for recovery instead of hammering a broken dependency.

Output
CLOSED (normal):  ingest -> transform -> publish
OPEN (tripped):   ingest fails quality gate -> STOP publish
                  alert owners; do not refresh marts
HALF-OPEN:        allow a probe run after fix / cooldown

Data examples that trip the breaker

  • Freshness breach beyond critical threshold
  • Row count drop of 90% vs baseline
  • Contract / schema validation failure
  • Reconciliation mismatch above tolerance

Why it matters

Publishing a half-empty fct_orders can be worse than serving slightly stale yesterday data. Executive dashboards and ML features will amplify the bad batch.

Interview tip: "Circuit breaker = stop the blast radius when quality is critically bad." Contrast with silent publish + hope someone notices.