Overview
One group, many consumers. Partitions split round-robin so each aisle has one reader in the group.
On this page8 sections
The picture
ShopKart's order topic has 3 partitions, and 90,000 events a second arrive, 30,000 in each lane. One worker can handle 30,000 a second. So one worker keeps up with one lane and falls behind on the other two.
The fix is to hire more workers and give each worker one lane. Workers that share a topic this way form a consumer group.
A group coordinator assigns each partition to one consumer. If worker c2 crashes, the coordinator triggers a rebalance and hands partition 1 to worker c1, so every lane keeps moving.
Members of one group share partitions. 2 groups each get their own view of the full topic.
Left side: one group splitting the partitions to share the load. Right side: two independent groups, each receiving every message on the topic.
A single order event can feed a warehouse ingestion pipeline and a real-time fraud alert without either consumer slowing or starving the other.

Words you will see in this lesson
Six words. The rest of the lesson uses only these.
| Word | What it means |
|---|---|
| Consumer group | Readers that share the work of a topic |
| Rebalance | Reassigning lanes when a reader joins or leaves |
| Lag | How far behind the reader is |
| Partition | One lane of a topic |
| Worker | One reader in the group |
| Throughput | How much data per second |
Why this shape
Two fundamental rules govern every consumer group.
- Inside one group, each partition is read by exactly one worker. That is what stops 2 workers from processing the same message twice.
- Different groups are independent. Each group maintains its own bookmarks (offsets), so a slow analytics worker never holds back a live fraud detector.
The first rule has a direct consequence: you can never use more active workers than partitions. With 3 partitions and 4 workers, the fourth worker has nothing to read and sits idle.
| Partitions | Workers in the group | What happens |
|---|---|---|
| 3 | 1 | One worker reads all 3 lanes and falls behind |
| 3 | 2 | One worker reads 2 lanes, the other reads 1 |
| 3 | 3 | One lane each, the ideal fit |
| 3 | 4 | 3 workers read, the fourth is an idle standby |
Walk the boxes
The simplest way to distribute partitions is round-robin: deal them out like cards, one worker at a time. Divide the partition index by the number of consumers and take the remainder.
Round-robin: consumers[i % len(consumers)]
| Partition index | Partition | Assigned consumer (round-robin with 2 consumers) |
|---|---|---|
| 0 | p0 | c1 (0 % 2 = 0) |
| 1 | p1 | c2 (1 % 2 = 1) |
| 2 | p2 | c1 (2 % 2 = 0) |
A matching example
Predict first: with 3 partitions and 2 workers, which worker gets partition p2?
def assign_partitions(partitions, consumers):
"""Assign partitions to consumers using round-robin."""
assignment = {c: [] for c in consumers}
for i, partition in enumerate(partitions):
consumer = consumers[i % len(consumers)]
assignment[consumer].append(partition)
return assignment
# 3 partitions, 2 consumers
result = assign_partitions(["p0", "p1", "p2"], ["c1", "c2"])
print(result)
# 3 partitions, 3 consumers (perfect split)
result2 = assign_partitions(["p0", "p1", "p2"], ["c1", "c2", "c3"])
print(result2){'c1': ['p0', 'p2'], 'c2': ['p1']}
{'c1': ['p0'], 'c2': ['p1'], 'c3': ['p2']}Consumer c1 wraps around and also gets p2. With 3 workers, each worker receives exactly 1 partition.
What happens when a worker leaves or joins?
The group reassigns the lanes. This process is called a rebalance. If c2 crashes, c1 inherits its lane. If a fourth worker joins with only 3 partitions, it gets nothing.
# What happens when a consumer leaves?
assignment_after_crash = assign_partitions(["p0", "p1", "p2"], ["c1"])
print("After c2 crash:", assignment_after_crash)
# What happens when more consumers than partitions?
assignment_idle = assign_partitions(["p0", "p1", "p2"], ["c1", "c2", "c3", "c4"])
print("4 consumers:", assignment_idle)After c2 crash: {'c1': ['p0', 'p1', 'p2']}
4 consumers: {'c1': ['p0'], 'c2': ['p1'], 'c3': ['p2'], 'c4': []}Consumer c1 shoulders all 3 partitions after the crash. When 4 workers join, consumer c4 holds an empty list: an idle standby ready to take over if another worker fails.
How do you know the group is keeping up?
Check the lag: the newest offset in a partition minus the group's committed offset. If lag continuously rises, incoming traffic outpaces consumer capacity, dashboards fall behind, and messages risk deletion if they exceed retention limits.
What if the warehouse is slower than the topic?
Suppose the warehouse accepts only 200 rows a second, but workers read 2,000. In-memory buffers fill rapidly. This situation is backpressure, and the consumer must regulate its ingestion rate.
- Pause reading from the partition. The worker remains in the group and retains its partition ownership.
- Flush buffered rows to the warehouse at a pace the destination can accept.
- Resume reading once buffer capacity frees up.
Reading smaller batches also helps, such as fetching 50 messages at a time instead of 500. If a worker takes too long processing one batch, the group coordinator assumes it died and reassigns its partitions.
Common beginner questions
What happens if I have more consumers than partitions?
The extra consumers sit idle as standbys. If an active consumer crashes, the coordinator reassigns an orphaned partition to an idle worker. Scaling beyond the partition count provides redundancy, not higher throughput.
Can 2 consumers in the same group read the same partition?
No. Within a group, each partition is assigned to at most one consumer to prevent duplicate processing. If two applications need the same data, assign them different group IDs.
What is the group.id?
It is a string identifier for the consumer group, such as 'gold-builder' or 'fraud-detector'. All consumers sharing that group.id divide topic partitions among themselves.
Consumers with group.id = 'gold-builder' share work among themselves. Consumers with group.id = 'fraud-watch' form an independent team with separate offsets. The two teams never interfere with each other's messages.
If you configure 5 consumers for a 3-partition topic, 2 consumers will have nothing to do. Scale consumers up to the partition count, not beyond.
Traffic: ShopKart's orders topic has 3 partitions and 90,000 events a second. One worker handles 30,000, so one worker is not enough.
- 90,000 / sec
- 1 worker
- falls behind
Coordinator: 2 workers join the group gold-builder. The group deals the lanes out: c1 gets p0 and p2, c2 gets p1.
- c1: p0, p2
- c2: p1
Rebalance: c2 crashes. The group re-deals and c1 now reads all three lanes, slower but nothing is skipped.
- c2 crash
- c1 takes p1
Second group: A fraud team starts a second group, fraud-watch. It reads every message too, with its own bookmarks, and never slows the first group.
- new group
- reads all
- own offsets
Monitor: Lag tells the on-call engineer whether any group is falling behind. A fourth worker would be idle, because there are only 3 lanes.
- watch lag
- max workers = 3
What comes next
Consumer groups split the work cleanly across partitions. But what happens when a consumer crashes after processing a record but before committing the offset? The next lesson covers delivery guarantees, offset commits, and at-least-once delivery.
Practice
Implement assign_partitions(['p0', 'p1', 'p2'], ['c1', 'c2']). c1 should get p0 and p2, c2 should get p1.
Going deeper: consumer groups, rebalances and duplicates
A consumer group splits one topic between several copies of the same program. The part that surprises people is not the splitting itself but what happens when members come and go: work is paused, partitions change owners, and uncommitted records are handled twice.
Rule one: a partition has one owner per group at a time
Within a group, each partition is read by exactly one member. That is what preserves order within a partition. It also sets a hard ceiling on parallelism: more members than partitions leaves the extras idle.
6 partitions, and members join and leave. The assignment below uses simple round-robin. Real assignors differ in the exact split, but the shape is the same.
def assign(parts, members):
ms = sorted(members)
out = {m: [] for m in ms}
for i, p in enumerate(sorted(parts)):
out[ms[i % len(ms)]].append(p)
return out
for members in (["A"], ["A","B"], ["A","B","C"], ["A","B"], list("ABCDEFG")):
print(members, assign([0,1,2,3,4,5], members))Predict the last line before you reveal it: which member gets no partitions?
['A'] {'A': [0, 1, 2, 3, 4, 5]}
['A', 'B'] {'A': [0, 2, 4], 'B': [1, 3, 5]}
['A', 'B', 'C'] {'A': [0, 3], 'B': [1, 4], 'C': [2, 5]}
['A', 'B'] {'A': [0, 2, 4], 'B': [1, 3, 5]}
['A', 'B', 'C', 'D', 'E', 'F', 'G'] {'A': [0], 'B': [1], 'C': [2], 'D': [3], 'E': [4], 'F': [5], 'G': []}Look at the last line: member G owns nothing. Seven members on 6 partitions means one is always idle.
What triggers a rebalance
| Trigger | What the coordinator saw | How to avoid needless ones |
|---|---|---|
| A member joins | A new JoinGroup request | Roll deployments gradually, or use static membership (a fixed group.instance.id) so a quick restart keeps its partitions |
| A member leaves | A clean shutdown | Expected; nothing to fix |
| A member goes silent | No heartbeat within session.timeout.ms | Do not starve the heartbeat thread; check for long garbage-collection pauses |
| A member is too slow | No call to poll() within max.poll.interval.ms | Process fewer records per poll (max.poll.records), or speed up the handler |
| Partitions change | Topic metadata changed | Plan partition changes; they move key ranges |
Trace a rebalance with a duplicate
Two members, A and B, share partitions 0 and 1. A has handled offsets 20 to 24 of partition 0 but has not committed yet when it is kicked out for a slow poll. Follow the timeline.
Offsets 20 to 24 were processed twice.
| Time | Event | Partition 0 owner | Committed offset |
|---|---|---|---|
| t0 | A is handling offsets 20 to 24, last commit is 20 | A | 20 |
| t1 | A's handler is slow; poll() is not called within max.poll.interval.ms | A | 20 |
| t2 | Coordinator removes A and starts a rebalance; everyone pauses fetching | none | 20 |
| t3 | New generation: partition 0 is assigned to B | B | 20 |
| t4 | B starts at the committed offset, 20, and handles 20 to 24 again | B | 20 then 25 |
| t5 | A finishes and tries to commit 25 with the old generation id | B | rejected |
- A slow, kicked out
- rebalance pauses reads
- B takes partition 0
- B restarts at last commit
- records 20 to 24 repeat
Two protections cut in here. First, the generation id on every commit makes the coordinator reject a commit from a member that no longer owns the partition, so the stale A cannot move the bookmark backwards or forwards by mistake.
Second, the repeat of 20 to 24 is the normal at-least-once replay, and an idempotent sink makes it harmless.
With the older eager protocol every member gives up all its partitions during a rebalance, so everything pauses. With the cooperative-sticky assignor members keep partitions that do not need to move, and only the moving partitions pause. The duplicate trap above is the same either way: the moved partition restarts from its last committed offset.
Try a new scenario
A topic has 12 partitions and a group of 4 consumers. You scale to 16 consumers because lag is growing. Predict what happens to throughput and what you would change instead.
Answer, then open it
Only 12 consumers can own a partition, so 4 sit idle and throughput stops improving at 12. Lag stays high. Either the partition count must grow (which moves keys and changes the order of old and new events for the same key), or each consumer must handle its partition faster. Check first whether the handler is the bottleneck, for example one slow database call per record.
Why does a slow handler cause duplicates and not just lag?
If one poll batch takes longer than max.poll.interval.ms, the coordinator treats the member as dead and reassigns its partitions. The new owner replays everything since the last commit while the old member may still be running. Smaller batches and faster handlers keep the member inside the time limit.
Close the lesson and explain it
Without scrolling up, explain why a rebalance can create duplicates. Then answer: what changes if you set max.poll.records much lower, and what do you pay for it?
Check yourself
Each poll returns fewer records, so each batch finishes sooner and the member is less likely to exceed max.poll.interval.ms, so fewer needless rebalances. The price is more poll calls and commits, which adds some overhead per record.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.
Practice this
Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.