System design interview

Lambda vs Kappa Architecture

EasyFree75 min read

Explain Lambda (batch + speed layers) vs Kappa (streaming-only), when to recommend each, and Lambda's code-divergence failure mode.

system-designlambdakappaarchitecture-patterns

Start here

The problem in plain words

Your company counts daily active users. Every morning a batch job finishes and says 1.04 million. During the day, a live dashboard fed by a streaming job says 990,000 for the same day. A vice president asks which number is real.

Neither of them lied. Two teams wrote the idea of an "active user" twice. One counts a user as active for a 24-hour session and removes bots using the warehouse. The other uses a 30-minute inactivity timeout and a bot list copied into the code. Two code bases, two answers.

Lambda and Kappa are two names for two answers to that mess. Lambda keeps a slow, careful batch path and a fast, rougher live path side by side. Kappa keeps one path, and fixes mistakes by replaying the stored events through the same code.

Words you will meet

Batch processing
Reading a large slice of stored data and computing results on a schedule, such as every night. Slow but thorough.
Stream processing
Computing results continuously as each event arrives. Fast, but you may see late or incomplete data.
Event log
A durable, ordered record of everything that happened, such as a Kafka topic. Stored events can be read again later.
Replay
Reading old events again from the log to recompute results, for example after fixing a bug.
Lambda architecture
A batch layer for correct history, a speed layer for recent data, and a serving layer that answers queries from both.
Kappa architecture
A single streaming pipeline over a retained log. Corrections are done by replaying that log through new code.
Code divergence
Two implementations of the same business rule slowly drifting apart, so their results disagree.
Late data
Events that arrive after the time they describe, for example from a phone that was offline.
Two paths, two answers to "how many active users?"
nightlyresultliveresultRaw eventsclicks and opensBatch jobSQL, 24-hour sessionStreaming job30-minute timeout1.04 Myesterday's report990 Klive dashboard
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Every night the batch job recomputes active users with warehouse SQL and a bot list from the warehouse.
  2. 2.It says 1.04 million, and finance trusts it.
  3. 3.Meanwhile a separate streaming job counts the same events as they arrive, with its own rules.
  4. 4.It says 990 thousand. Different rules, different bot list, different handling of late events, so a different answer.

The numbers, as things you can picture

  • Tens of thousands of events a secondFor example, 20,000 a second is about 1.7 billion events a day. Big for one machine, ordinary for a streaming system.
  • A few data engineersA small team can maintain one pipeline well, but two parallel pipelines double the work and the risk.

What the interviewer wants

Explain Lambda and Kappa clearly enough for a hiring manager and deeply enough for a senior engineer, give a recommendation for a mid-size team, and name the main failure of Lambda.

Interview framing

You are in a mid-level data engineering interview. The interviewer asks you to compare Lambda Architecture and Kappa Architecture for analytics pipelines.

They are not fishing for buzzwords. They want to know whether you have lived the pain of two pipelines that disagree, whether you understand replay, and whether you can recommend something a mid-size team can operate without drowning.

A strong answer starts with the product problem that created these patterns. Then you name the layers. Then you tell the failure story. Then you pick for *this* company.

Background: what existed before Lambda

Before "Lambda Architecture" was a phrase, most analytics looked like this.

Events and logs landed in files or a database dump. Overnight, a batch job scanned yesterday's data and wrote daily aggregates into a warehouse. Product managers opened dashboards at 8 AM and saw yesterday's truth.

That model was correct for many questions. It was also late. Product started asking for "active users in the last 5 minutes," "error rate right now," and "did the checkout funnel break after the deploy." Waiting until tomorrow was not acceptable.

Teams bolted on a second path: a streaming job that approximated the same metrics in seconds. The warehouse stayed the "official" numbers. The stream fed live dashboards. For a while everyone was happy.

Then the numbers diverged. Batch said 1.02M daily actives. Stream said 980K. Finance trusted batch. Ops trusted stream. Nobody could explain the gap in a sentence. That disagreement is the historical problem Lambda tried to organize, and it is still the scoring center of this interview question.

Background: first principles before the names

You need three ideas before you draw boxes.

1. Correctness vs freshness

Some questions need exact, reproducible answers over large history (revenue by day with late refunds applied). Some questions need a fresh approximate answer (errors in the last 60 seconds). One engine and one schedule rarely optimize both.

2. Two implementations of the same business rule

If you compute "active user" in SQL with one set of window rules, and again in a streaming job with different event-time semantics, you have two products pretending to be one metric. Drift is not a bug in Kafka. Drift is a process failure.

3. Replay as a first-class operation

When you discover a bug in how you counted, can you rebuild last week's metrics from durable raw inputs? If yes, you can fix history. If no, you live with wrong dashboards forever or with painful one-off backfills.

Lambda and Kappa are two different answers to how you split correctness, freshness, and replay.

Full problem statement

Compare Lambda and Kappa for a mid-size company:

  • Tens of thousands of events per second (not hyperscale).
  • A few data engineers maintain pipelines.
  • Product wants some live metrics and some trusted daily numbers.
  • Clickstream is a fine concrete domain: page views, clicks, purchases.

Explain:

  • Lambda's batch layer, speed layer, and serving layer.
  • Kappa's single streaming pipeline plus replay from a durable log.
  • When you recommend each.
  • What goes wrong with Lambda in practice, especially code divergence.
  • How corrections and backfills work in each.

What "good" looks like in the room

You tell a short story about disagreeing metrics before you define terms. You draw both architectures on the board. You walk a clickstream example through each path. You spend real time on code divergence with a vivid failure. You discuss replay cost. You make a recommendation tied to team size and whether a durable log with weeks of retention is feasible. You do not declare Kappa always better.

Clarifying questions to ask

  • Do we already have a mature batch warehouse and SQL culture?
  • Is a durable, replayable log (Kafka or similar) already in place, and how long is retention?
  • How often do metric definitions change?
  • Latency needs: seconds vs minutes vs hours?
  • Who is the consumer of "truth" for finance vs ops dashboards?
  • Can the streaming engine economically replay 30-90 days of history?

Scale and estimation prompts

Say rough numbers out loud even if the prompt is qualitative.

  • 50K events/sec × 500 bytes ≈ 25 MB/sec ≈ ~2 TB/day raw before compression.
  • Kafka with 3× replication multiplies network and disk.
  • Replaying 30 days at catch-up speed may need far more compute than the live job.
  • Serving-layer merge in Lambda adds read complexity: how do apps combine batch truth and speed approximations?

What is out of scope

  • Picking a specific vendor logo as the answer (Kafka vs Pulsar, Flink vs Spark) without tying it to constraints.
  • Designing a full multi-region DR plan unless asked.
  • Deep algorithm design for exact distincts at hyperscale.
  • Claiming one architecture is always modern and the other is obsolete.

Keep the conversation on layers, divergence, replay, and a defendable recommendation for a mid-size DE team.