Day 40Phase 09 · Streaming + CDC
Change data capture basics
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Explain why Change Data Capture (CDC) via database WAL outperforms scheduled nightly dump queries.
- Parse standard CDC event envelopes (before, after, op, timestamp) produced by Debezium.
- Apply an out-of-order stream of CDC mutations (INSERT, UPDATE, DELETE) to a warehouse table using MERGE.
What is Change Data Capture, in One Picture?
You have two ways to find out what changed in a database. You can query it periodically, or you can listen to its internal transaction journal.
Every relational database writes its intentions to a journal before updating data files. In PostgreSQL, this journal is the write-ahead log (WAL).
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Change Data Capture (CDC) via database write-ahead log (WAL / binlog), Debezium connector architecture & event envelopes (before, after, op), Out-of-order events & watermark event-time processing, Deduplication & exactly-once processing guarantees, Streaming mutations and tombstones, Changelog table compaction
Build: CDC stream processing engine: PostgreSQL WAL log parser → Debezium event envelope flattener → out-of-order watermark buffer → compacted lakehouse sink.
Never query source databases directly for real-time changes. Read the database WAL via Debezium to avoid impacting transactional query performance.
Already have a pass? Go to your account. Need a pass? See pricing.