System design interview

Handle Late-Arriving Data in Streaming

MediumLocked55 min read

Strategy for late data: watermarks, allowed lateness, nightly reprocessing, append-only design with read-time dedupe.

system-designstreamingwatermarkslate-data

Start here

The problem in plain words

A user's phone was in airplane mode for two hours. When it reconnects, it sends all the events it saved. Your streaming job, meanwhile, has already closed the 10:00 to 11:00 window and shipped that hour's numbers to a dashboard. Now an event stamped 10:50 turns up at 13:10. What do you do with it?

If you have no answer, you do not have a design for late data. You have hope. There is always some data that arrives late, from phones, from other regions, from slow networks, and numbers that silently ignore it drift away from the truth.

A late-data policy is something you choose on purpose. A common choice is to wait a little while in the live stream, accept that the live number is approximate, and then fix the final number overnight from the raw data.

Words you will meet

Event time
When the event actually happened, as stamped by the device.
Window
A slice of time, such as 10:00 to 11:00, over which a result is computed.
Watermark
The stream job's best guess that events up to some time have all arrived, used to decide when to close a window.
Allowed lateness
Extra waiting time after the watermark, during which late events can still change a window's result.
Late event
An event that arrives after its window has already closed.
Side output
A separate stream where too-late events are sent, so they are kept instead of dropped.
Raw lake
Cheap storage that keeps every raw event, so you can recompute results later.
Reprocessing
Running the calculation again over stored raw data, now including the late events.
Append-only with dedupe at read
Never edit old rows. Add corrected rows, and pick the latest version when reading.
The event that arrived after its window closed
13:10publishedPhone offlineevent at 10:50Stream job10:00 to 11:00 alreadyclosedDashboardhour already published
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.The phone reconnects at 13:10 and sends an event that happened at 10:50. The 10:00 to 11:00 window closed long ago.
  2. 2.The hour's revenue was already sent to the dashboard. If the late event is ignored, the number stays a little too low.

The numbers, as things you can picture

  • An event 2 hours latePhones in airplane mode or tunnels are normal, so lateness of minutes or hours is a design input, not an exception.

What the interviewer wants

Give a complete late-data policy that covers the stream and a batch correction path, bounds state growth, and makes it clear to consumers which numbers are final.

The interview setup

Design a strategy for late-arriving data in streaming. Discuss watermarks (allowed lateness), reprocessing (nightly batch), and append-only design with deduplication at read time.

Late data is not a Flink footnote. It is a product policy question: when a phone was in airplane mode for 2 hours, do you revise the dashboard, ignore the events, or land corrections beside immutable closes?

Late means "after we thought we were done"

A window for hour 10 closes in your stream job. An event with event_time inside hour 10 arrives at wall-clock 13:00. That event is late relative to your watermark decision.

Watermarks and allowed lateness

Watermarks estimate event-time progress. Allowed lateness keeps windows open a bit longer for updates. Then windows close and further events need another policy.

Batch reconcile

Many teams use stream for speed and nightly batch from raw logs for truth. Late events naturally appear in the batch rebuild.

Append-only facts with read-time resolution

Store every observation with versions. Views pick the latest per key. Writers stay simple. Readers become smarter. This pattern shows up in ledgers and some lakehouse designs.

The expanded problem

Provide a coherent late-data policy that spans:

  • Stream watermarks and allowed lateness
  • Side outputs for very late events
  • Nightly or on-demand reprocessing
  • Append-only + dedupe-at-read as an alternative
  • Communication of correctness expectations to consumers
Stream windows (fast, may revise within lateness)
Raw log --> nightly job --> curated daily tables (truth)

What good looks like

  • Airplane-mode story up front
  • Explicit product policy
  • Watermark + lateness with state bound
  • Batch truth path
  • Answer for immutable finance closes

Clarifying questions

  • How late is late for this business: minutes, hours, days?
  • Can downstream tolerate revisions?
  • Do finance closes need immutability?
  • What generates lateness: mobile, multi-region, upstream batch?

Out of scope

  • Perfect prediction of all late distributions
  • Building a new stream engine
  • Ignoring business policy in favor of only tooling

How to use clarifying questions

Ask a few questions that change architecture: SLA definition, source of truth, retention, and tolerance for approximates. If told to decide, state assumptions on the board and proceed.

What good sounds like

Narrate trade-offs while drawing. Name the waiting room for bursts, the transform, the serving surface, and the replay path. Explicitly list out-of-scope items.

Before you draw

Find entry, buffer, transform, serve, and replay. If any is missing, your operational story has a hole.