Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Stream-table duality

Kafka · Operations & Scenarios

Stream-table duality

Mediumkafka-58
stream-table-dualitykafka-streamslog-compactioncdc

Question

What is stream-table duality?

Solution

Stream-table duality is the principle that an event stream and a relational table are two representations of the same underlying data: a stream represents a changelog stream recording every historical mutation, while a table represents the current state computed by aggregating that changelog. You can reconstruct a table at any point by replaying the changelog stream from the beginning, and you can generate a stream by capturing every insert, update, and delete executed on a table. Kafka implements this relationship through log compaction, Change Data Capture (CDC), and Kafka Streams abstractions like KStream and KTable.

The two perspectives on data

Consider how customer account balances evolve over time:

Changelog Stream:
Offset 0: (user_42, balance = 100)
Offset 1: (user_42, balance = 250)
Offset 2: (user_42, balance = 180)

State Table:
Key user_42 -> current balance = 180

A short summary clarifies how these two representations interact:

  • The stream preserves the full immutable history of actions, answering what happened and when.
  • The table collapses history into a snapshot of the latest state, answering what the current value is right now.
  • Applying a stream to a blank slate creates a table; logging the mutations of a table produces a stream.

Implementations in Kafka Streams

In the Kafka Streams library, this duality is expressed through two primary abstractions:

  • KStream: Models an append-only stream where every incoming record is an independent event, such as individual credit card transactions.
  • KTable: Models an evolving changelog where incoming records represent upserts, and a record with a null value represents a tombstone that deletes the key.

Physical storage: log compaction and CDC

Kafka brokers physically implement the table side of this duality using log-compacted topics. In a compacted topic, the background cleaner purges obsolete offsets for each key, keeping only the most recent value on disk. Change Data Capture tools like Debezium apply stream-table duality directly to databases, converting relational tables into changelog streams that downstream applications replay to rebuild synchronized local caches.

PreviousNext