Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing

Streaming & Message Queues

Progress0/13
x

Batch versus Streaming

  • Batch versus streaming12m
  • Streaming architectures12m
  • Exactly-once semantics14m

Why Queues

  • Why a queue12m
  • Topics, partitions, keys14m

Offsets and Groups

  • Offsets and commits14m
  • Consumer groups14m

Delivery and Time

  • At-least-once14m
  • Late and out of order14m
  • Watermarks and Spark14m

Stream-batch

  • Stream-batch unification12m
  • Replay from offset12m

Capstone

  • Capstone: late messages16m
Back to track
  1. Learn
  2. Streaming & Message Queues
  3. Delivery and Time
  4. Watermarks and Spark

Lesson 10 of 13 · Case study

Watermarks and Spark

streamingpythonintermediate14 min

Overview

Advance the door as max event minutes minus a delay. Cousin of Structured Streaming, not a copy of that exercise.

Module: Delivery and Time

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

This lesson requires Pro

This lesson is part of Delivery and Time. Pro opens the full lesson and the exercises.

Compare Free vs Pro