Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Batch + streaming together

Data platform · Platform Concepts

Batch + streaming together

Hardplatform-24
hybridlambdastreamingbatcharchitecture

Question

How can a platform combine batch and streaming without chaos?

Solution

Most real platforms are hybrid. Streaming handles low-latency paths; batch handles completeness, heavy transforms, and reconciliation.

A clean pattern: Lambda-ish or staged hybrid

stream path:  events -> Kafka -> near-real-time table (fast, maybe approximate)
batch path:   events -> lake raw -> daily rebuild gold (complete, trusted)
serve layer:  BI reads gold; ops dashboards may read stream table

Rules that keep you sane

  • One raw system of record for replay (lake or retained log)
  • Same business keys and contracts across paths
  • Clear which table is "source of truth" for metrics
  • Periodic reconciliation between stream and batch totals

Interview tip: Do not claim everything must be streaming. Describe a fast path + a trusted batch path and how you reconcile them.

PreviousNext