Kafka · Extra High-Value

Why can't you scale consumers infinitely in a group?

Mediumconsumer grouppartitionsscalabilityparallelism

Interview question

Why does adding more consumers in the same group eventually stop helping?

How to answer

The short answer

Because parallelism inside one is capped by the number of .

The hard rule

In a group, each partition is assigned to at most one consumer.

So:

Adding a 7th consumer does not increase throughput for that group. It just sits unused (until another leaves or you add partitions).

Diagram

Output
Topic (4 partitions)

Group with 4 consumers: each busy
Group with 8 consumers: 4 busy, 4 idle

How to scale further

  1. 1Increase partitions (plan carefully: keys remap when count changes; ordering across old/new layout needs thought).
  2. 2Make each consumer faster (batching, async IO, better sink).
  3. 3Split work across different groups only if you need independent fan-out (that reads the full stream again; it does not split one group's load).
  • "We'll add 100 consumers" on a 3-partition topic → 97 idle.
  • Increasing partitions helps throughput but can break assumptions if you depended on a specific key→partition mapping forever.

Interview tip: "Consumer group parallelism ≤ partition count. Idle consumers mean you need more partitions or faster processing, not more of the same group members."