Batch & Streaming · Streaming Processing

Lambda architecture

MediumLambdabatchspeed layerarchitecture

Interview question

What is the Lambda architecture?

How to answer

The short answer

Lambda architecture keeps two paths to answer queries on the same data:

  1. 1Batch layer: periodic recomputation over the full master dataset (accurate, slower)
  2. 2Speed layer: streaming approximations for recent data (fast, may be incomplete)
  3. 3Serving layer: merges batch views + real-time views for the user
Output
                 +--> batch (nightly Spark) ----+
all events -----+                               +--> serving merge --> query
                 +--> speed (stream) -----------+

Why it existed

Early streaming engines were weaker; batch was trusted for correctness while stream filled the freshness gap.

Pain

You maintain two codepaths for similar business logic. Bugs diverge. Ops cost doubles.

Interview tip: Define batch + speed + serving, then say many teams moved toward Kappa or unified batch/stream APIs to avoid dual logic.