System design interview

Real-Time Analytics Pipeline (2M Events/Second)

HardFree75 min read

Design a pipeline that ingests 2M mobile events/sec (~1KB each), processes them in real time, and serves dashboards with a 10-second refresh SLA.

system-designstreamingreal-time-analyticsestimation

Start here

The problem in plain words

Picture a food-delivery app on millions of phones. Every tap, every screen opened and every order placed sends a tiny message to the company's servers saying "this just happened". The product team wants one live board that shows how the app is doing right now, and it must never be more than 10 seconds behind real life.

The first idea most people have is the obvious one: save every message into a normal database and let the board ask the database questions. That works for a small app. At this app's size, 2 million messages arrive every second, and a normal database simply cannot write that fast or answer questions over that much data while it is still arriving.

So the real job is to design a small chain of systems. One catches the flood, one turns the flood into a few running totals, one serves those totals to the board, and one keeps a cheap raw copy for later. The skill being tested is knowing why each piece exists, and doing the arithmetic before drawing anything.

Words you will meet

Event
One small message saying that something happened, such as "user 7 opened the menu at 10:02:11". An event is a fact and is never edited afterwards.
Throughput
How much data a system handles per second. Here that is events per second and gigabytes per second.
Freshness
How old the numbers on the board are. "10-second freshness" means the board is at most 10 seconds behind reality.
Message log (Kafka)
A durable queue where new messages are added to the end and readers read in order, each remembering its own place. Kafka is the best-known one. It lets producers and readers work at different speeds.
Partition
One lane of the log. Splitting the log into lanes lets many workers read at the same time. A key, such as the event type, decides which lane a message goes to.
Stream processing
A program that reads messages as they arrive and keeps running results, instead of waiting for a whole day of data. Flink and Spark Structured Streaming are common choices.
Aggregate
A summary of many events, such as "4,210 purchases in the last 10 seconds". One small aggregate row replaces thousands of events.
Serving store
A fast database that holds the ready-made summaries so the board can read them in milliseconds. Redis, Druid and ClickHouse are examples.
Data lake
Cheap storage that holds raw files, such as Parquet files in Amazon S3. Slow to query, but huge and inexpensive.
Consumer lag
How far a reader has fallen behind the newest message. If lag grows, the board gets stale.
Dead-letter queue
A side queue for messages that are broken or cannot be understood, so one bad message does not block everything behind it.
At-least-once delivery
A message is guaranteed to arrive, but might arrive twice after a retry. The receiver must cope with duplicates.
Why one database cannot be the answer
2M a secasktoo slowMillions ofphonestap, scroll, orderOne databasetries to save everyeventLive boardneeds fresh numbers
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Two million small messages arrive every second, all day. The database has to store each one.
  2. 2.The board asks, "how many purchases in the last 10 seconds?". To answer, the database must scan the newest events while more keep pouring in.
  3. 3.The writes and the questions fight each other. The database falls behind, the board goes stale and the whole system can fall over.

The numbers, as things you can picture

  • 2 million events a secondAbout the population of a large city, each person sending one message, every single second.
  • 1 KB per event, about 2 GB a secondEach message is about a short paragraph of text. Together they are about 2 GB every second. That is a lot more than a normal single database is built to absorb.
  • About 173 TB a day2 GB a second for 86,400 seconds. It would fill about 170 laptops with 1 TB disks every day, before any compression.
  • A dashboard that is 10 seconds old at mostFew people read the board, perhaps hundreds at a time. Reads are tiny. The write side is the hard part.

What the interviewer wants

Show the pipeline from phone to dashboard, say the byte maths out loud first, and explain what each box is for. You should also say how you handle spikes, bad messages, late messages and the replay of old data.

Interview framing

You are in a mid-level data engineering system design interview. Budget about 45 to 55 minutes. The interviewer will not grade you on naming a specific vendor. They want to hear how you think under load: clarify, estimate, draw a path that can fail safely, and defend trade-offs with numbers.

The prompt you will hear is short on purpose:

Design a system that ingests 2 million events per second from mobile apps, processes them in real time, and serves dashboards with a 10-second refresh SLA. Events are about 1 KB each.

That is enough to fail if you jump straight to boxes. Good candidates slow down, do byte math out loud, and only then sketch architecture.

Background from first principles

Start with what an event is. A mobile app fires a small JSON payload when something happens: screen view, button tap, purchase, crash. Each payload is roughly 1 KB. At 2 million such payloads every second, you are not talking about a web form. You are talking about a firehose.

Why does that matter? Write path and read path are different animals. Dashboards refresh every few seconds and serve a modest number of analysts. The write path must absorb millions of inserts per second without melting. If you treat both paths as one database, you lose.

Postgres and most OLTP databases shine at transactions, joins, and strong consistency for application state. They are not built to ingest 2 GB of new rows every second forever. Indexes, WAL, vacuum, and disk IOPS will collapse long before you hit the dashboard SLA. Saying that out loud is interview gold: we cannot write raw events into Postgres at this rate.

You need three ideas before the whiteboard:

1. A durable buffer that absorbs spikes (a log such as Kafka or a managed equivalent). 2. Stream processing that turns raw events into small aggregates on the fly. 3. A serving store optimized for fast dashboard reads, plus a cheap archive for raw history.

Also understand event time versus processing time. Event time is when the phone says the action happened. Processing time is when your job sees the record. Mobile networks make those diverge. Any serious real-time design must name a late-event policy.

Expanded problem statement and constraints

Build a first production version of a real-time analytics pipeline:

  • Peak ingest: 2,000,000 events/sec from mobile clients.
  • Payload size: about 1 KB per event.
  • Dashboard freshness: charts should not show data older than about 10 seconds at the serving layer. State your interpretation of end-to-end latency versus refresh interval.
  • Metrics: rolling counts, rates, approximate uniques, simple funnels suitable for ops and product.
  • Replay: keep a recent window of raw events so you can fix bugs and backfill.
  • Poison messages: bad payloads must not stall the hot path; route them to a dead-letter path.
  • Region: single-region v1 is acceptable if you name the DR gap.
  • Cost: you must defend storing and moving roughly 2 GB/sec of raw data.

What good looks like in the room

You restate the goal in one sentence. You write the byte math on the board before drawing Kafka. You ask which metrics truly need 10-second freshness versus nightly batch. You separate hot aggregates from cold raw archive. You name one late-event strategy and one lag alert. You leave time for failure modes. You do not promise exactly-once everywhere without an idempotent sink story.

The interviewer should leave believing you felt the scale, protected the product under spikes, and knew where truth lives when the hot path is approximate.

Clarifying questions to ask

  • Which metrics must be real-time vs can wait for a nightly batch?
  • Is uniqueness exact or approximate (HLL)? Does cross-device identity matter?
  • Retention for raw events vs aggregates?
  • Is the 10-second SLA end-to-end (phone to UI) or chart refreshes every 10s with whatever is already stored?
  • Multi-tenant customer dashboards or one internal BI surface?
  • Peak multiplier over average (2x to 3x is common)?
  • PII in the event payload (user ids, device tokens)?
  • Which dimensions are allowed on hot tiles (event_type, country) versus only in the lake (user_id)?

Scale and estimation prompts

Do this math out loud:

  • 2e6 events/sec x 1 KB = 2 GB/sec raw ingest.
  • Per day: 2 GB/sec x 86,400 sec is about 173 TB/day uncompressed.
  • With 3x to 5x columnar compression on cold storage, plan roughly 35 to 60 TB/day retained.
  • Kafka with 3x replication moves about 6 GB/sec of network traffic to brokers at peak.
  • Dashboard QPS is low (tens to hundreds). Optimize the write path into the serving store, not a giant scan of raw events every refresh.

Ask the interviewer whether peak is 2x or 3x average, and size the buffer and stream jobs for peak.

A useful verbal check: if your design still works if event size were 200 bytes or 5 KB, say which hop changes. That shows you tied architecture to numbers.

Out of scope for v1

Say these out loud so the interviewer knows you can bound the problem:

  • Multi-region active-active serving.
  • Exact distinct users over 30 days at full fidelity for every tile.
  • Real-time personalization or online feature serving for ML (different problem).
  • Guaranteed end-to-end exactly-once with zero cost (prefer at-least-once plus idempotent aggregates).
  • Storing every raw byte forever without tiering.
  • Full-text search over every raw event on the hot path.