Lesson 2 of 3 in this module

4/13 in track

Offsets and commits

python14 min

Overview

Each partition is a list. The offset is the next index. Commit is writing that number down.

On this page8 sections

The idea

It is 3 AM and a worker is counting ShopKart clicks. It has read 50 million messages over 10 hours when the server reboots for a security update.

When it comes back it has two bad options. Start again from the first message and spend another 10 hours on records it already counted. Or jump to the newest message and lose everything that arrived during the reboot.

To recover cleanly, the worker needs a bookmark that records where it stopped. When it comes back online, it reads the bookmark, resumes from message 50,000,000, and carries on without missing a click or redoing 10 hours of counting.

That bookmark is an offset. An offset is an integer marking the position of a message inside a partition. Offset 0 is the first message. If you read 3 messages starting at 0, your next offset is 3.

Committing an offset means saving that bookmark to durable storage, so a restarted consumer opens at the right place. In the Python editor in this tab, an offset is simply an integer index into a Python list.

An offset is a bookmark. A commit is putting the bookmark back in the book.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
OffsetThe position of an event in a partition
CommitRecord how far you have read
PartitionOne lane of a topic
ConsumerWhatever reads the events
CrashThe reader stops unexpectedly
BookmarkAnother way to picture an offset

Why this exists

Think of a reader working through 1,000-page manual. Dinner is called, so they slip a bookmark in at page 314. The next morning they open straight to page 314. Without that bookmark, they must reread from page 1 or guess where they stopped.

When you commit decides what a crash costs

A worker reads a message, writes it to the warehouse, and commits the offset. The order of those steps matters. Follow what happens if the worker crashes between the write and the commit.

Losing a message and repeating a message are the two failure modes of every queue.

OrderCrash in the middle meansYou get
Commit first, then write to the warehouseBookmark moved, row never writtenA lost message
Write to the warehouse first, then commitRow written, bookmark not movedThe same message read twice

Most pipelines choose the second row: a repeated message can be deduplicated, but a lost payment is permanently gone. The next module covers delivery guarantees in depth.

Offsets belong to a lane

A topic with 4 partitions has four separate offset counters, each starting at 0. Offset 42 in partition 0 and offset 42 in partition 3 are two unrelated messages. A bookmark always pairs both values: partition 2, offset 42.

Reading is not committing

A consumer's current read position moves forward in memory every time it fetches a batch. The committed offset only moves when the consumer writes that position to durable storage.

If the process crashes, the in-memory pointer is lost and the restart resumes from the last committed offset.

Picture this

Messages, offset, committed bookmark
Partition: [m0, m1, m2, m3, m4]Current offset: 3 (next to read)Committed offset: 2 (last saved)

The partition is a list of messages. The offset is your position. Committing saves that position.

Reading advances your local pointer in memory. Committing writes that offset to the broker's internal storage, so it survives a crash.

Trace the bookmark

The partition keeps every message in order. Your offset is the bookmark. Predict what a crash replays.

A small example

Predict first: the partition holds 5 messages and each read asks for two. What offset do you land on after each read?

Python
# Offsets as list indices
PARTITION = ["m0", "m1", "m2", "m3", "m4"]

def next_offset(messages, start, n):
    """Read n messages from start and return the next offset."""
    batch = messages[start:start + n]
    print(f"Read: {batch}")
    return start + len(batch)

# Read 2 messages starting from offset 0
offset = next_offset(PARTITION, start=0, n=2)
print(f"Next offset: {offset}")

# Read 2 more messages from offset 2
offset = next_offset(PARTITION, start=offset, n=2)
print(f"Next offset: {offset}")

# Read 2 from offset 4, but only 1 message remains
offset = next_offset(PARTITION, start=offset, n=2)
print(f"Next offset: {offset}")
Read: ['m0', 'm1']
Next offset: 2
Read: ['m2', 'm3']
Next offset: 4
Read: ['m4']
Next offset: 5

The last read asked for 2 messages, but only one remained at offset 4. The offset advanced by 1 to reach 5. An offset never moves past the end of the partition.

Now separate reading from committing. The consumer reads a batch, processes each record, and only commits after processing succeeds.

Python
class OffsetTracker:
    """Track committed offsets for a consumer."""
    def __init__(self):
        self.committed = 0

    def consume(self, partition, n):
        """Read n messages from the committed offset."""
        batch = partition[self.committed:self.committed + n]
        return batch

    def commit(self, new_offset):
        """Save the offset as committed (persisted on restart)."""
        self.committed = new_offset

tracker = OffsetTracker()
batch = tracker.consume(PARTITION, 2)
print(f"Processing: {batch}")
tracker.commit(2)

batch = tracker.consume(PARTITION, 2)
print(f"Processing: {batch}")
# A crash here leaves committed at 2, so restart re-reads m2 and m3
tracker.commit(4)
Processing: ['m0', 'm1']
Processing: ['m2', 'm3']

If a crash happens before tracker.commit(4), the saved bookmark remains at 2. On restart, tracker.consume(PARTITION, 2) reads m2 and m3 again. Nothing was lost, but 2 messages are repeated.

ActionEffect on offsetPurpose
ReadAdvances local in-memory pointerFetch records for the worker to process
CommitPersists the pointer to durable storageEnable clean restart without reprocessing earlier data
SeekMoves pointer to a specific offsetReplay past records or skip ahead

How often should you commit?

Committing requires a network call to the broker, so you trade safety against throughput.

  • After every message: safest, but slowest, because every single message waits for a network round trip to persist its bookmark.
  • After every batch (such as 500 rows written to the warehouse): the standard production choice. One network save per 500 records, and a crash replays at most one batch.
  • Asynchronously without waiting for broker confirmation: fastest, but a delayed retry can overwrite a newer offset with an older one.

What if the consumer has no bookmark at all?

A brand new consumer group, or one whose saved offset was deleted after aging out of retention, must pick a starting point. The configuration is auto.offset.reset.

SettingStarts atUse it when
earliestThe oldest message still kept in the partitionA pipeline needs historical data, as in a backfill or initial load
latestOnly messages that arrive from now onYou only care about live events, as in an alerting service
noneRaises an exception and haltsA missing bookmark indicates an operational issue that requires investigation

How do you tell a consumer is behind?

Consumer lag is the newest offset in a partition minus the consumer group's committed offset. If partition 2 is at offset 850,000 and the group has committed 820,000, the lag is 30,000 messages.

Lag holding steady near zero means consumers keep pace with incoming traffic. Lag climbing during peak traffic hours means workers are slower than incoming volume, so add consumers or optimize warehouse writes.

Engineers can also reset a group's offset using administrative tools, such as moving the bookmark back 4 hours to replay records after a bug fix.

Common beginner questions

Is the log destroyed after reading?

No. Unlike a traditional queue where consumed items are popped and deleted, Kafka retains messages for a configured retention period. Reading does not delete anything. This is what makes replay possible: a consumer can seek back to any offset within the retention window.

What happens if I never commit?

If the consumer restarts, it reprocesses everything from offset 0 (or from its last committed offset). This duplicates work, but leaves data correct if downstream writes are idempotent.

Can 2 consumers have different offsets on the same partition?

Yes, if they belong to different consumer groups. Each consumer group maintains its own independent offset tracking for every partition.

RememberBookmarks in data engineering

In Change Data Capture (CDC, see [CDC to raw landing](/learn/orchestration-pipelines/orch-cdc)), database replication uses Log Sequence Numbers (LSNs) as bookmarks. Kafka offsets follow the exact same principle: a sequential integer marking progress so a consumer can restart safely after failure.

One reboot, told with a bookmark
  1. Worker: A worker is counting clicks. It has read up to offset 50,000,000 and has committed that bookmark.

    1. read
    2. process
    3. commit 50,000,000
  2. Broker: At 3 AM the server reboots. The worker's memory is gone, but the bookmark is safe in the broker.

    1. reboot
    2. memory lost
    3. bookmark kept
  3. Worker: The worker asks for its bookmark and resumes from offset 50,000,000. It does not start over and it does not skip ahead.

    1. ask bookmark
    2. resume at 50,000,000
  4. Worker: Because it commits after writing, the worst case is that one small batch is read twice. Nothing is lost.

    1. write row
    2. then commit
  5. Monitor: Lag shows whether the worker keeps up. The gap between the newest offset and the bookmark is the number to watch.

    1. newest offset
    2. minus bookmark
    3. equals lag

What comes next

A single consumer reading from 1 partition is fine for moderate throughput. To scale out across multiple partitions, multiple consumers join together in a consumer group. That is the next lesson, on consumer groups and rebalances.

Practice

Implement next_offset(messages, start, n) on PARTITION from offset 0 taking 2 messages. Result is 2.

Going deeper: commits, crashes and replay

Offsets feel simple until the first crash. This section pins down three details that cause real incidents: what number a commit actually stores, how often you commit, and what a consumer does when it has no saved position at all.

The number you commit is the next offset, not the last one

A partition holds records numbered 0, 1, 2 and so on. If you have finished the record at offset 6, the position you save is 7: the next record to read. This off-by-one is the classic bug: commit 6 and your consumer re-reads record 6 after every restart.

Committed position means 'start here after a restart'.

OffsetRecordStatus once offset 6 is finished
0 to 6HandledBehind the bookmark
7Next unread recordCommitted position = 7
8 and 9Not yet readAhead of the bookmark

Trace a crash with a commit every 5 records

Committing after every record is expensive, so consumers commit in batches. 10 records, a commit after every five handled, and a crash while the consumer is about to handle offset 7. Predict which offsets are handled twice before you read the output.

PythonBatch commits and a crash
log = list(range(10))          # offsets 0..9 in 1 partition
committed = 0
processed = []

def consume(crash_at=None, batch=5):
    global committed
    pos, since = committed, 0
    while pos < len(log):
        if pos == crash_at:
            return "crashed at offset %d" % pos
        processed.append(pos)
        pos += 1
        since += 1
        if since == batch:         # commit the NEXT offset to read
            committed, since = pos, 0
    committed = pos
    return "done"

print(consume(crash_at=7), "committed =", committed)
print(consume(), "committed =", committed)
print("processed order:", processed)
print("lag before restart:", len(log) - 5)

Which two offsets do you expect to see twice in the processed list?

crashed at offset 7 committed = 5
done committed = 10
processed order: [0, 1, 2, 3, 4, 5, 6, 5, 6, 7, 8, 9]
lag before restart: 5

Offsets 5 and 6 were handled before the crash but not yet committed, so they are handled again. The size of the replay is always 'records since the last commit'. A bigger batch means fewer commits and a larger replay; a smaller batch means more commits and a smaller replay. Neither is free.

  1. handle 5
  2. handle 6
  3. crash before 7
  4. restart at 5
  1. handle 5 and 6 again

What if there is no saved position?

A brand-new consumer group has never committed anything. The setting auto.offset.reset decides where it starts. It only applies when there is no valid committed position (a new group, or the saved offset no longer exists because the data aged out of retention).

auto.offset.resetA new group starts atUse it whenRisk
earliestThe oldest record still retainedA pipeline must see all history, such as a new warehouse sinkA long catch-up, and a flood of old events
latestRecords that arrive after the consumer joinsA live alert only cares about nowSilently skips everything produced while the consumer was down for the first time
noneNothing. It raises an errorYou want a human to decideThe job fails until someone sets a position
Watch outAuto-commit commits what was fetched, not what you finished

With enable.auto.commit=true a background timer saves the position of records already handed to your code. If your code is still writing them to the warehouse when it crashes, the position has moved past records that never landed. For a sink you care about, commit manually after the write succeeds.

Lag in one line

Consumer lag per partition is the log-end offset minus the committed offset. In the trace above, the end is 10 and the commit is 5, so lag is 5. Lag that stays near zero means the consumer keeps up.

Lag that grows steadily means it is slower than the producers. A sudden spike followed by recovery usually means a restart or a rebalance.

Try a new scenario

A partition holds offsets 0 to 19. A consumer commits after every 4 records and crashes just before handling offset 11. Where does the restarted consumer start, which offsets are handled twice, and what is the lag at the moment of the crash?

Work it out, then open the answer

Commits happen after offsets 3, 7 and 11 would be handled, so the saved position values are 4, 8, 12. The consumer handled 0 to 10 but only reached the commit at position 8, so the saved position is 8. Restart begins at offset 8; offsets 8, 9 and 10 are handled twice. Lag at the crash is 20 minus 8, which is 12 (the log end offset is 20).

A replayed record writes a duplicate row. Is the offset logic broken?

No. Replay after a crash is the designed behaviour of at-least-once processing. The fix belongs in the sink: an upsert or unique key on the record's business id.

Close the lesson and explain it

Without scrolling up, explain what the committed offset means and why the replay size depends on the commit interval. Then answer: if a consumer commits every 1,000 records and crashes, what is the most it can re-handle after restart?

Check yourself

Up to 1,000 records, because the last commit could be at most 1,000 handled records old. If the sink is idempotent those repeats change nothing.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Practice this

Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.

Rate:
Was this useful?