Batch & Streaming · Core Concepts

Keyed vs non-keyed streams

Mediumkeyed streamstateparallelismskew

Interview question

What is the difference between keyed and non-keyed stream processing?

How to answer

The short answer

A keyed stream events by a key (user_id, device_id, order_id). Operators maintain per-key state, and events with the same key are processed in order on one parallel instance.

A non-keyed (or global) stream has no key. Aggregations are global, parallelism is limited, and state is shared carefully (often a single parallelism of 1 for correctness).

Output
Keyed by user_id
  key=u1 --> operator instance A   (state for u1, u4, ...)
  key=u2 --> operator instance B
  key=u3 --> operator instance A

Non-keyed global count
  all events --> one logical aggregator (harder to scale)

Why keys matter

  • Parallelism: different keys on different workers
  • Correctness: per-user sessions need keyed state
  • Hot keys: one celebrity user_id can overload one worker (skew)

Interview tip: "Key = how we shard state and parallelism." Contrast with Kafka partitions: similar idea of ordered shards.