Kafka · Producers & Consumers

Consumers and offsets

Easyconsumeroffsetcommit__consumer_offsets

Interview question

How do Kafka consumers use offsets to track progress?

How to answer

The short answer

An offset is a monotonically increasing ID for each record inside a . It is the consumer's bookmark.

Output
partition 0:
offset:  0     1     2     3     4
record: [A]   [B]   [C]   [D]   [E]
                 ^
          consumer last committed = 1
          next fetch starts at 2

Who stores offsets?

For , Kafka stores committed offsets in an internal topic usually named __consumer_offsets (keyed by group + topic + partition).

Commit styles

  • Auto commit (enable.auto.commit=true): periodic background commits. Easy, but can cause duplicates or skips depending on timing vs processing.
  • Manual commit: app commits after successful processing (commitSync / commitAsync). More control.

The pattern that matters

Commit too early and you risk at-most-once (skipping work). Crash before commit and you get at-least-once redelivery.

Independent progress

Different consumer groups each keep their own offsets for the same topic. One group at offset 1000 does not block another still at offset 10.

Interview tip: "Offset = position in a partition log. Committing says: I am done up to here."