Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Consumers and offsets

Kafka · Producers & Consumers

Consumers and offsets

Easykafka-15
consumeroffsetcommit__consumer_offsets

Question

How do Kafka consumers use offsets to track progress?

Solution

An offset is a monotonically increasing ID for each record inside a partition. It is the consumer's bookmark.

partition 0:
offset:  0     1     2     3     4
record: [A]   [B]   [C]   [D]   [E]
                 ^
          consumer last committed = 1
          next fetch starts at 2

Who stores offsets?

For consumer groups, Kafka stores committed offsets in an internal topic usually named __consumer_offsets (keyed by group + topic + partition).

Commit styles

  • Auto commit (enable.auto.commit=true): periodic background commits. Easy, but can cause duplicates or skips depending on timing vs processing.
  • Manual commit: app commits after successful processing (commitSync / commitAsync). More control.

The pattern that matters

poll → process records → commit offsets

Commit too early and you risk at-most-once (skipping work). Crash before commit and you get at-least-once redelivery.

Independent progress

Different consumer groups each keep their own offsets for the same topic. One group at offset 1000 does not block another still at offset 10.

Interview tip: "Offset = position in a partition log. Committing says: I am done up to here."

PreviousNext