Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing

Streaming & Message Queues

Progress0/13
x

Batch versus Streaming

  • Batch versus streaming12m
  • Streaming architectures12m
  • Exactly-once semantics14m

Why Queues

  • Why a queue12m
  • Topics, partitions, keys14m

Offsets and Groups

  • Offsets and commits14m
  • Consumer groups14m

Delivery and Time

  • At-least-once14m
  • Late and out of order14m
  • Watermarks and Spark14m

Stream-batch

  • Stream-batch unification12m
  • Replay from offset12m

Capstone

  • Capstone: late messages16m
Back to track
  1. Learn
  2. Streaming & Message Queues
  3. Stream-batch
  4. Stream-batch unification

Lesson 11 of 13 · Case study

Stream-batch unification

streamingpythonintermediate12 min

Overview

A nightly GMV job is a bounded stream. A click topic is unbounded. Same engine idea, different label.

Module: Stream-batch

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

This lesson requires Pro

This lesson is part of Stream-batch. Pro opens the full lesson and the exercises.

Compare Free vs Pro