Pipelines & scenarios · Design & Scenarios

How do you monitor a pipeline?

MediummonitoringalertingobservabilityfreshnessDQ

Interview question

How do you monitor a data pipeline in production?

How to answer

The short answer

Monitoring answers: Did it run? Is data fresh? Is it correct enough? Is the system healthy?

Output
Layers of monitoring
  1) Orchestration: task success/fail, duration, SLA miss
  2) Data: row counts, null rates, freshness, anomalies
  3) Platform: Kafka lag, Spark stage time, warehouse slots/cost
  4) Business: revenue reconciliation vs payments source

Useful signals

  • Job failure / retry spikes
  • Data arrival time vs
  • Empty or missing
  • Volume spikes/drops vs yesterday/week-ago baseline
  • Consumer lag (streaming)
  • DQ test failures

Alerting hygiene

Page humans for SLA risk and data corruption risk. Ticket noisy warnings. Include runbook links.

Tools (examples)

Airflow/Dagster UI, metrics (Prometheus/CloudWatch), DQ (dbt tests, Great Expectations), lineage (OpenLineage), log search.

Interview tip: Structure as pipeline health + data health + platform health. Mention freshness and row-count anomaly checks explicitly.