Consumer group offsets are stored inside an internal, log-compacted Kafka topic named __consumer_offsets. One specific broker, acting as the group coordinator for a given consumer group, manages that group's commits and writes them as keyed messages containing the consumer group ID, topic name, and partition number. This internal architecture replaced legacy ZooKeeper offset tracking, and offsets for inactive groups automatically expire after offsets.retention.minutes, forcing restarted consumers to use auto.offset.reset.
How the offsets topic is structured
By default, __consumer_offsets has 50 partitions and a replication factor of 3. When a consumer group commits an offset, Kafka computes which partition manages that group using a simple hash:
partition = Math.abs(group.id.hashCode()) % 50
A short explanation clarifies how leadership determines the coordinator:
- The broker hosting the leader replica of that assigned partition becomes the group coordinator for that consumer group.
- Every commit writes a record where the key is group ID, topic, and partition, and the value contains the committed offset, leader epoch, and commit timestamp.
- Because the topic uses log compaction, Kafka continuously cleans old offset records and keeps only the latest committed offset for each key.
Inactivity expiration and reset pitfalls
The broker configuration offsets.retention.minutes (default 7 days in modern versions) determines how long Kafka retains committed offsets for a group after all members disconnect.
If a consumer service is shut down for maintenance or fails for longer than this retention window, the group coordinator deletes its offset entries during log compaction. When the consumer application boots back up, Kafka treats it as a brand-new group with no commit history. The consumer must then rely on auto.offset.reset, either skipping unread data if set to latest or reprocessing all retained topic history if set to earliest.