Phase 04 · Modern data platforms · streaming
Streaming & Message Queues
- lessons
- 13
- modules
- 6
- stages
- 6
- of learning
- 2h 54m
Concept Traces in this track
Playable walkthroughs: watch the system move, predict the next step, stamp a memory seal, then practice. Completing a Trace counts toward readiness.
Why this track exists
A fraud team needs to block a suspicious card in seconds. A nightly batch job tells them tomorrow, which is useless. So the data has to move continuously: every transaction, as it happens. That changes everything about how you handle it. There is no end of file, so you never know if you have seen everything. A network hiccup means a message might arrive twice. An event from a phone with bad signal might arrive 20 minutes late, after you already reported the total. Streaming systems are built around those three facts.
Fewer jobs are pure streaming than job ads suggest, but the vocabulary shows up constantly: topics, partitions, offsets, consumer groups, exactly once, watermarks. Interviews ask about it even for batch roles, because the reasoning about duplicates and late data applies everywhere.
What you need before starting
Streaming is best understood as a contrast with batch. If you have never built a batch job, do that first.
Producers and consumers here are Python, simulated in the browser. No broker to install.
The roadmap
6 stages, in the order they build on each other. Each stage lists the modules and lessons it covers, and what you should be able to do by the end of it.
01Batch versus streaming, and why a queue
Start with when streaming is genuinely worth it, because it costs more to build and operate than batch. Most data does not need it. Then see why a queue sits between the program that sends events and the one that reads them.
Batch versus StreamingFree
0/2
When batch is enough, when you need real-time, and why a message queue exists.
By the end of this stage
You can decide whether a requirement actually needs streaming, and explain what a queue buys you.
02Partitions, offsets, and consumer groups
This is the mechanical core. Ordering, parallelism, and how a consumer knows where it left off after a crash all come from these three ideas.
Partitions, Offsets and GroupsFree
0/3
How FakeKafkaTopic splits keys across partitions, a bookmark per partition, then 2 consumers in one group splitting the aisles.
By the end of this stage
You can explain how work is split across consumers, what ordering guarantee you actually get, and what happens when a consumer dies mid-batch.
Where people get stuck
Ordering is per partition, not global. Most streaming misunderstandings trace back to missing that.
03Delivery promises
At least once, at most once, exactly once. These promises decide what happens to a message when a consumer crashes, and they are the topic people most often get wrong.
Delivery PromisesFree preview
0/2
At-least-once vs a commit, then why exactly-once is hard.
By the end of this stage
You can explain what each promise guarantees, why a commit matters, and why exactly once is hard.
Where people get stuck
Exactly once is widely misunderstood. Learn what it actually guarantees and where the boundary is.
04Late data and watermarks
Event time versus processing time, and what to do with events that arrive late. These are the topics that separate people who have run a stream from people who have read about one.
Time
0/2
Late event_time, then a watermark that is not the PySpark exercise.
By the end of this stage
You can explain what a watermark is for, and what tradeoff you make when you choose how long to wait for late events.
05Architectures, and streaming with batch
Real systems are both. The interesting question is how to keep a streaming view and a batch view from disagreeing.
Architectures and Replay
0/3
Lambda and Kappa designs, batch as a bounded stream, and replay as seek-and-reread on the same FakeKafkaTopic.
By the end of this stage
You can describe how a real architecture serves fresh data and correct historical data at the same time.
06Capstone
Put the pieces together on a realistic event flow with duplicates and late arrivals.
Capstone
0/1
Produce three FakeKafkaTopic records, drop the late one, keep ORD-2 and ORD-3.
By the end of this stage
You can design a consumer that is safe to restart and honest about lateness.
Finish this track, get a certificate
Complete all 13 lessons and claim a certificate with a link you can add to LinkedIn or send to a recruiter.
Mistakes to avoid on this track
| Common mistake | What to do instead |
|---|---|
| Assuming exactly once means duplicates are impossible everywhere. | It is a guarantee about a specific boundary. Your sink still usually needs idempotent writes. |
| Committing the offset before the work is done. | A crash then loses the message silently. Commit after the write, and make the write safe to repeat. |
| Choosing streaming because it sounds more advanced. | Streaming doubles operational burden. If the business can wait an hour, batch is the better engineering decision. |
Where to practise this
Where to go after this
Streaming & Message Queues reviews & rating
Finished a few lessons? Tell us what worked and what was confusing. Reviews are read by hand and help decide what gets rewritten next.