Day 39Phase 09 · Streaming + CDC

Streaming basics

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Explain distributed commit logs: topics, partitions, offsets, and consumer groups.
  • Route messages deterministically using message keys and explain partition ordering guarantees.
  • Compare delivery semantics (at-least-once, at-most-once, exactly-once) and identify duplicate risk.

The Problem: When Batch Processing Is Too Slow

In Days 36 through 38, you built a Medallion batch pipeline for ShopKart. That pipeline runs on a scheduled interval, such as every morning at 05:00 UTC or once every hour. Scheduled batch runs are ideal for daily revenue reconciliations, financial reporting, and executive business dashboards.

Now consider what happens when a customer card is compromised, or a shopper adds high-value electronics to their cart and closes the tab. If your fraud detection service waits for the 05:00 UTC batch run, the fraudulent order has already cleared and shipped from the fulfillment center. If your marketing automation service waits 24 hours to trigger an abandoned cart reminder, the customer has already purchased the item on a rival website.

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: Distributed commit log architecture, Brokers, topics, partitions & segments, Message keys & deterministic partition routing, Producers: acks (0, 1, all) & idempotence, Consumer groups, rebalancing & offset commit semantics, Lag monitoring & consumer scale limits

Build: Kafka streaming simulator: producer partition router with hashing, multi-consumer group coordinator, offset commit manager, and lag spike monitor.

Partition count is the fundamental unit of parallelism. You cannot have more active consumers in a consumer group than partitions in a topic.

Already have a pass? Go to your account. Need a pass? See pricing.