Day 40Phase 09 · Streaming + CDC

Change data capture basics

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Explain why Change Data Capture (CDC) via database WAL outperforms scheduled nightly dump queries.
  • Parse standard CDC event envelopes (before, after, op, timestamp) produced by Debezium.
  • Apply an out-of-order stream of CDC mutations (INSERT, UPDATE, DELETE) to a warehouse table using MERGE.

What is Change Data Capture, in One Picture?

You have two ways to find out what changed in a database. You can query it periodically, or you can listen to its internal transaction journal.

Every relational database writes its intentions to a journal before updating data files. In PostgreSQL, this journal is the write-ahead log (WAL).

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: Change Data Capture (CDC) via database write-ahead log (WAL / binlog), Debezium connector architecture & event envelopes (before, after, op), Out-of-order events & watermark event-time processing, Deduplication & exactly-once processing guarantees, Streaming mutations and tombstones, Changelog table compaction

Build: CDC stream processing engine: PostgreSQL WAL log parser → Debezium event envelope flattener → out-of-order watermark buffer → compacted lakehouse sink.

Never query source databases directly for real-time changes. Read the database WAL via Debezium to avoid impacting transactional query performance.

Already have a pass? Go to your account. Need a pass? See pricing.