Kafka · Extra High-Value

Why do we use a partition key?

Easypartition keyorderingpartitionerskew

Interview question

Why do Kafka producers set a message key, and how does it affect partitioning?

How to answer

The short answer

The key (when present) typically decides which a record goes to:

Output
partition = hash(key) % numPartitions

(Exact partitioner details can vary, but this is the interview model.)

Why keys matter

  1. 1Ordering: all records with the same key land in the same partition → processed in order for that key.
  2. 2Colocation: related events stay together (e.g. all updates for user_id=42).
  3. 3Compaction: compacted topics need keys to keep "latest value per key."

Examples of good keys

  • order_id for order lifecycle events
  • customer_id for customer profile updates
  • account_id for balance changelogs

Examples of bad key choices

  • Random keys when you needed per-entity order
  • Always-null keys when you needed compaction
  • Extremely hot keys (one celebrity user_id) creating a hot partition

Skew warning

If 80% of traffic shares one key, 1 partition (and one consumer) becomes the bottleneck. Choose keys for both correctness and balance.

Interview tip: "Keys give you per-entity ordering and enable compaction; watch for hot keys."