Kafka guarantees strict message ordering only within individual partitions, not across an entire multi-partition topic. Achieving absolute global ordering across a full topic requires restricting that topic to exactly one partition, which sacrifices horizontal scalability and limits consumption to a single worker instance. In practice, data systems almost never need total global order; partitioning by an entity key like order_id or user_id preserves strict chronological ordering for each entity while distributing throughput across many partitions.
The cost of global ordering
When you configure a topic with one partition:
- Only one broker handles all read and write traffic for that topic, bottlenecking cluster throughput.
- Only one consumer instance in any consumer group can process records; adding more consumers results in idle standby pods.
- If that single consumer experiences high CPU or slow database writes, the entire pipeline backs up with growing lag.
Total ordering across unrelated events is rarely a genuine business requirement. An e-commerce platform does not care whether order 501 is processed before or after order 502 from a different customer; it only requires that updates for order 501 arrive in strict sequence.
Achieving per-entity ordering with keys
By supplying an entity identifier as the record key:
- The default partitioner hashes the key and consistently routes all events for that entity to the exact same partition.
- Each partition processes its assigned slice of entities in strict sequential order.
- You achieve horizontal scaling across dozens of partitions while maintaining causal ordering per entity.
Protecting ordering during producer retries
Network timeouts can silently break ordering if retries are not configured properly. If batch 1 fails transiently and batch 2 succeeds on the broker, retrying batch 1 will append it after batch 2. To prevent this, enable the idempotent producer (enable.idempotence=true, default in modern Kafka). Idempotent producers track sequence numbers per batch on the broker, allowing you to run up to max.in.flight.requests.per.connection=5 without risking message reordering or duplication on retry.