System design interview

Monitor Production Pipeline Health

MediumLocked55 min read

5-layer monitoring: job health, freshness, volume, quality, infrastructure (Kafka lag, Spark memory).

system-designobservabilitymonitoringops

Start here

The problem in plain words

The scheduler shows every job green. Then finance says the revenue report is wrong. The jobs really did run, but one of them read only a fifth of the usual data. Everything looked healthy, because the only thing being watched was whether the jobs ran.

A pipeline can fail in many ways that a green tick does not show: the machines can be struggling, the job can finish with partial data, the data can be late, the volume can be far too low, or the values can be wrong. To trust a pipeline, you have to look at all of those, and not only at the scheduler.

A useful way to organise it is five layers, from the machines at the bottom to the correctness of the data at the top. You also connect the layers, so that when something breaks you can tell which layer is the cause.

Words you will meet

Observability
Being able to tell from the outside what is happening inside a system, through metrics, logs and checks.
Infrastructure metrics
Health of the machines and platforms: Kafka consumer lag, Spark memory, disk and CPU.
Job health
Whether jobs ran, how long they took and whether they failed.
Volume
How many rows arrived, compared with what is normal.
Freshness
How recent the newest data is.
Data quality
Whether values are right: no unexpected nulls, no duplicates, totals that make sense.
Baseline
What normal looks like for a metric, so abnormal stands out.
Runbook
A short document that tells the on-call person what to check and do for a given alert.
RACI
A chart of who is responsible, accountable, consulted and informed.
Green on the scheduler, wrong in the report
successwrongSchedulerall jobs greenSource datavolume down 80%Finance reportrevenue too low
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.The job finishes without errors, so the scheduler says success. It simply processed very little input.
  2. 2.The finance report is built from that small input and shows far less revenue. No alarm rang, and a person has to notice it.

The numbers, as things you can picture

  • Several critical pipelines on shared Kafka and SparkA problem in a shared platform can affect many pipelines at once, so you need to see the platform layer too.

What the interviewer wants

Design monitoring in layers so that a green job cannot hide bad data, and so that you can tell which layer a problem comes from.

Interview room

5-layer monitoring: job, freshness, volume, quality, infra. Green Airflow is not healthy data.

Basics

List 5 layers with example metrics. Alerts need owners and runbooks. Layers catch empty success and silent staleness.

Prompt

Metrics, routing, dashboards, runbooks, avoid ownerless explosion.

Good

Five headings first; correlation; tier-0 pane.

Clarify/scope

On-call topology; tier-0 pipeline.

Scoring lens: Monitor Production Pipeline Health

Interviewers reward incident-first teaching, a clear control loop, and named owners. Tool logos without that loop score poorly.

Weak vs strong answers on Monitor Production Pipeline Health

Weak: list products. Strong: tell the failure story, define the mechanism, draw the path, state rollout and monitors.

Scope control

Say what is out of scope for v1. Come back only if follow-ups demand it. This is a seniority signal for Monitor Production Pipeline Health.

Clarifying then committing

Ask a few high-use questions about SLAs, owners, and scale, then sketch. Endless questions without a diagram look evasive.

Single pane for on-call

Tier-0 pipeline view: job status, freshness, volume band, last DQ, Kafka lag, memory. Five apps is how incidents get longer.

Inhibit rules

When parent source freshness is already P1, inhibit duplicate child pages or attach them as related. Storm control is design, not luck.

Metric ownership tax

Adding a page requires an owner field. Ownerless metrics are deleted in monthly reviews.

Interview framing detail 1

For lb-sd-34, spend the first minutes making the problem concrete: who gets hurt when this fails, what SLA is implied, and what is explicitly out of scope. Then teach the prerequisite concept before proposing boxes. A candidate who names vendors first usually loses the plot. Write two clarifying questions on the board and answer them with assumptions if the interviewer shrugs. Keep the first diagram small enough to redraw when a follow-up changes a constraint.