Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Why can't you scale consumers infinitely in a group?

Kafka · Extra High-Value

Why can't you scale consumers infinitely in a group?

Mediumkafka-33
consumer grouppartitionsscalabilityparallelism

Question

Why does adding more consumers in the same group eventually stop helping?

Solution

Because parallelism inside one consumer group is capped by the number of partitions.

The hard rule

In a group, each partition is assigned to at most one consumer.

So:

partitions = 6
consumers  = 3  → ~2 partitions each → useful
consumers  = 6  → 1 partition each  → max parallelism
consumers  = 7  → one consumer idle (no partition left)

Adding a 7th consumer does not increase throughput for that group. It just sits unused (until another leaves or you add partitions).

Diagram

Topic (4 partitions)

Group with 4 consumers: each busy
Group with 8 consumers: 4 busy, 4 idle

How to scale further

1. Increase partitions (plan carefully: keys remap when count changes; ordering across old/new layout needs thought). 2. Make each consumer faster (batching, async IO, better sink). 3. Split work across different groups only if you need independent fan-out (that reads the full stream again; it does not split one group's load).

Related interview traps

  • "We'll add 100 consumers" on a 3-partition topic → 97 idle.
  • Increasing partitions helps throughput but can break assumptions if you depended on a specific key→partition mapping forever.

Interview tip: "Consumer group parallelism ≤ partition count. Idle consumers mean you need more partitions or faster processing, not more of the same group members."

PreviousNext