Share groups, introduced in KIP-932 as Queues for Kafka, allow multiple consumers to pull and process records cooperatively from the same partition at the same time. Unlike standard consumer groups that assign each partition to a single consumer instance, share groups manage delivery and acknowledgments on a per-record basis. This enables Kafka topics to operate as point-to-point worker queues where consumer concurrency is not capped by partition count, though it removes strict message ordering guarantees.
Breaking the partition concurrency barrier
In a classic consumer group, if an orders topic has 8 partitions, a group can run at most 8 active consumers. Spinning up a ninth consumer leaves that instance completely idle.
Standard Consumer Group (1:1 Partition to Consumer):
Partition 0 --------> Consumer A (exclusively reads Partition 0)
Partition 1 --------> Consumer B (exclusively reads Partition 1)
Share Group (M:N Cooperative Queue):
Partition 0 ----+---> Consumer A (pulls record 1, acks record 1)
+---> Consumer B (pulls record 2, acks record 2)A clear look at how share groups change consumption:
- Multiple consumers subscribe to the same share group and pull records concurrently from any partition.
- If processing a message takes several seconds, other consumers continue reading subsequent records from that same partition.
- Teams can scale worker fleets to hundreds of instances without creating hundreds of topic partitions.
Per-record acknowledgment and ordering trade-off
Instead of advancing a single partition offset, share groups track the state of individual in-flight records. Consumers explicitly acknowledge records with accept, release, or reject outcomes. If a worker crashes or times out while processing a record, the broker releases that record and redelivers it to another healthy consumer.
Because independent consumers process records from the same partition in parallel at different speeds, strict in-order processing is lost. Share groups are designed for asynchronous tasks like image processing, email dispatching, or webhook delivery, where work-queue semantics matter more than sequence ordering. The feature is in early access in Kafka 4.0, so check cluster feature flags and client version support before adopting it in production.