Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. What is log compaction?

Kafka · Advanced Kafka

What is log compaction?

Mediumkafka-21
log compactionchangelogtombstonecdc

Question

What is Kafka log compaction, and when would you use it?

Solution

Log compaction retains at least the last known value for each key in a partition, instead of (or in addition to) deleting only by time/size.

Mental model

Think of a compacted topic as a changelog of a table keyed by primary key.

Before compaction (same key "user-1"):
  offset 1: user-1 → {city: Pune}
  offset 2: user-1 → {city: Mumbai}
  offset 3: user-2 → {city: Delhi}

After compaction:
  offset 2: user-1 → {city: Mumbai}   ← latest for user-1
  offset 3: user-2 → {city: Delhi}

Older values for user-1 can be removed. Latest per key remains.

Great use cases

  • CDC changelogs (Debezium-style)
  • Kafka Streams / ksqlDB state store backups (changelog topics)
  • "Current entity snapshot" streams (customers, account_status)

Tombstones

A record with value null for a key is a tombstone. Compaction eventually removes the key entirely after the delete retention window. That models deletes.

What compaction does not mean

  • It is not a full database.
  • Ordering still matters; compaction is per partition by key.
  • Keys must be meaningful; null keys are not useful for compaction semantics.

Interview tip: "Compaction keeps the latest value per key. Perfect for changelogs and materialized views, not for pure event history."

PreviousNext