Monitor freshness SLAs, volume anomalies, and P1/P2 alerting tiers.
system-designobservabilityfreshnessalerting
Start here
The problem in plain words
A manager opens the sales dashboard at 2 pm and the numbers look strange. The numbers are from yesterday. Everyone checks the pipeline, and the scheduler shows a green tick: the job ran successfully. But it ran on old input, and no new data was loaded.
A green job does not mean fresh data. The job can succeed and still deliver nothing new, because the source stopped sending, or a file never arrived. What you need to watch for is the absence of new truth.
So you monitor the data, not just the job. For each important table, you decide how fresh it must be, you measure how old its newest data is, and you alert when it falls behind. You also watch the volume, because a table can be fresh and still contain only a trickle of rows.
Words you will meet
Freshness
How recent the newest data in a table is.
Lag
The gap between now and the time of the latest data in the table.
SLA
A promise about freshness, such as "fact_orders is never more than 4 hours old".
Volume anomaly
A row count far from what is normal for that table and time, such as 10% of the usual rows.
Baseline
What normal looks like, often different on weekdays and weekends.
P1 and P2
Alert severities. P1 pages someone now. P2 creates a ticket for working hours.
Alert storm
Dozens of alerts at once, usually because one upstream problem makes every table below it late.
RACI
A simple chart of who is responsible, accountable, consulted and informed for a problem.
A green job, and yesterday's numbers
Press Play to watch it move
Each arrow is one thing happening. Play it step by step, or read the list below.
1.The job ran and finished successfully, but the source sent nothing new, so it loaded nothing.
2.The dashboard keeps showing yesterday's numbers all day, with nothing red anywhere to warn anyone.
The numbers, as things you can picture
Hundreds of tables with different SLAsA table for finance and a table for an experiment do not need the same freshness. One global rule fits neither.
What the interviewer wants
Design monitoring for how fresh and how complete data is, with alerts of different severity that tell the right person what to do, without flooding anyone.
Interview room
Monitor freshness SLAs, volume anomalies, P1/P2 tiers. Green jobs can still be stale.
Basics
Freshness is data time, not DAG success. SLA catalog per table. Volume baselines catch empty success. Severity prevents fatigue.
Prompt
Track watermarks vs SLA; volume bands; route P1/P2; runbooks; hundreds of tables.
Interviewers reward incident-first teaching, a clear control loop, and named owners. Tool logos without that loop score poorly.
Weak vs strong answers on Data Freshness Monitoring
Weak: list products. Strong: tell the failure story, define the mechanism, draw the path, state rollout and monitors.
Scope control
Say what is out of scope for v1. Come back only if follow-ups demand it. This is a seniority signal for Data Freshness Monitoring.
Clarifying then committing
Ask a few high-use questions about SLAs, owners, and scale, then sketch. Endless questions without a diagram look evasive.
Watermark policy note
Event tables: now - max(event_time). Snapshot tables: now - last_publish_ts. Mixing them silently creates fake pages. Document per table class.
Source-late RACI
When bronze watermark is stale, page source owners primary and DE secondary. DE should not be the only human punished for vendor outages.
Planned windows
Encode expected batch completion windows so midnight work does not look like an incident. Mute is not the same as ignore: windows are explicit.
Interview framing detail 1
For lb-sd-29, spend the first minutes making the problem concrete: who gets hurt when this fails, what SLA is implied, and what is explicitly out of scope. Then teach the prerequisite concept before proposing boxes. A candidate who names vendors first usually loses the plot. Write two clarifying questions on the board and answer them with assumptions if the interviewer shrugs. Keep the first diagram small enough to redraw when a follow-up changes a constraint.