Day 44 of 45 · Pro lesson

Capstone Project #2: Real-Time Platform

The curriculum for this day stays visible. The lesson, code, and workspace unlock with Pro.

Teaches: Real-time e-commerce architecture specification, PostgreSQL transactional source → Debezium CDC → Kafka event bus, PySpark Structured Streaming with watermarks & stateful deduplication, S3/ADLS Medallion lakehouse (Delta/Iceberg/Parquet), Warehouse dimension modeling & incremental dbt mart, Production monitoring, alerting & end-to-end data contracts

Build: Capstone Platform: PostgreSQL CDC → Kafka streaming pipeline → PySpark streaming enrichment → Medallion lakehouse → dbt analytical warehouse.

•Complete 15 drills on end-to-end contract testing, streaming checkpoints, compaction schedules, and warehouse reconciliation.
•Run 6 Python modules verifying streaming state recovery, watermark cutoff adherence, and end-to-end latency.
•Simulate 5 production failures: broker disconnect, checkpoint directory corruption, schema drift event, and warehouse connection timeout.
•Complete the 100% end-to-end capstone platform checklist and code verification.

Build every layer independently and verify contracts at each boundary. If a bug occurs, isolate whether the source is CDC, Kafka, Spark, or the warehouse.

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

Already on Pro? Go to your account. Need a pass? See pricing.