Overview
Each partition is a list. The offset is the next index. Commit is writing that number down.
On this page8 sections
The idea
It is 3 AM and a worker is counting ShopKart clicks. It has read 50 million messages over 10 hours when the server reboots for a security update.
When it comes back it has two bad options. Start again from the first message and spend another 10 hours on records it already counted. Or jump to the newest message and lose everything that arrived during the reboot.
To recover cleanly, the worker needs a bookmark that records where it stopped. When it comes back online, it reads the bookmark, resumes from message 50,000,000, and carries on without missing a click or redoing 10 hours of counting.
That bookmark is an offset. An offset is an integer marking the position of a message inside a partition. Offset 0 is the first message. If you read 3 messages starting at 0, your next offset is 3.
Committing an offset means saving that bookmark to durable storage, so a restarted consumer opens at the right place. In the Python editor in this tab, an offset is simply an integer index into a Python list.
An offset is a bookmark. A commit is putting the bookmark back in the book.
Words you will see in this lesson
Six words. The rest of the lesson uses only these.
| Word | What it means |
|---|---|
| Offset | The position of an event in a partition |
| Commit | Record how far you have read |
| Partition | One lane of a topic |
| Consumer | Whatever reads the events |
| Crash | The reader stops unexpectedly |
| Bookmark | Another way to picture an offset |
Why this exists
Think of a reader working through 1,000-page manual. Dinner is called, so they slip a bookmark in at page 314. The next morning they open straight to page 314. Without that bookmark, they must reread from page 1 or guess where they stopped.
When you commit decides what a crash costs
A worker reads a message, writes it to the warehouse, and commits the offset. The order of those steps matters. Follow what happens if the worker crashes between the write and the commit.
Losing a message and repeating a message are the two failure modes of every queue.
| Order | Crash in the middle means | You get |
|---|---|---|
| Commit first, then write to the warehouse | Bookmark moved, row never written | A lost message |
| Write to the warehouse first, then commit | Row written, bookmark not moved | The same message read twice |
Most pipelines choose the second row: a repeated message can be deduplicated, but a lost payment is permanently gone. The next module covers delivery guarantees in depth.
Offsets belong to a lane
A topic with 4 partitions has four separate offset counters, each starting at 0. Offset 42 in partition 0 and offset 42 in partition 3 are two unrelated messages. A bookmark always pairs both values: partition 2, offset 42.
Reading is not committing
A consumer's current read position moves forward in memory every time it fetches a batch. The committed offset only moves when the consumer writes that position to durable storage.
If the process crashes, the in-memory pointer is lost and the restart resumes from the last committed offset.
Picture this
The partition is a list of messages. The offset is your position. Committing saves that position.
Reading advances your local pointer in memory. Committing writes that offset to the broker's internal storage, so it survives a crash.
Trace the bookmark
The partition keeps every message in order. Your offset is the bookmark. Predict what a crash replays.
A small example
Predict first: the partition holds 5 messages and each read asks for two. What offset do you land on after each read?
# Offsets as list indices
PARTITION = ["m0", "m1", "m2", "m3", "m4"]
def next_offset(messages, start, n):
"""Read n messages from start and return the next offset."""
batch = messages[start:start + n]
print(f"Read: {batch}")
return start + len(batch)
# Read 2 messages starting from offset 0
offset = next_offset(PARTITION, start=0, n=2)
print(f"Next offset: {offset}")
# Read 2 more messages from offset 2
offset = next_offset(PARTITION, start=offset, n=2)
print(f"Next offset: {offset}")
# Read 2 from offset 4, but only 1 message remains
offset = next_offset(PARTITION, start=offset, n=2)
print(f"Next offset: {offset}")Read: ['m0', 'm1'] Next offset: 2 Read: ['m2', 'm3'] Next offset: 4 Read: ['m4'] Next offset: 5
The last read asked for 2 messages, but only one remained at offset 4. The offset advanced by 1 to reach 5. An offset never moves past the end of the partition.
Now separate reading from committing. The consumer reads a batch, processes each record, and only commits after processing succeeds.
class OffsetTracker:
"""Track committed offsets for a consumer."""
def __init__(self):
self.committed = 0
def consume(self, partition, n):
"""Read n messages from the committed offset."""
batch = partition[self.committed:self.committed + n]
return batch
def commit(self, new_offset):
"""Save the offset as committed (persisted on restart)."""
self.committed = new_offset
tracker = OffsetTracker()
batch = tracker.consume(PARTITION, 2)
print(f"Processing: {batch}")
tracker.commit(2)
batch = tracker.consume(PARTITION, 2)
print(f"Processing: {batch}")
# A crash here leaves committed at 2, so restart re-reads m2 and m3
tracker.commit(4)Processing: ['m0', 'm1'] Processing: ['m2', 'm3']
If a crash happens before tracker.commit(4), the saved bookmark remains at 2. On restart, tracker.consume(PARTITION, 2) reads m2 and m3 again. Nothing was lost, but 2 messages are repeated.
| Action | Effect on offset | Purpose |
|---|---|---|
| Read | Advances local in-memory pointer | Fetch records for the worker to process |
| Commit | Persists the pointer to durable storage | Enable clean restart without reprocessing earlier data |
| Seek | Moves pointer to a specific offset | Replay past records or skip ahead |
How often should you commit?
Committing requires a network call to the broker, so you trade safety against throughput.
- After every message: safest, but slowest, because every single message waits for a network round trip to persist its bookmark.
- After every batch (such as 500 rows written to the warehouse): the standard production choice. One network save per 500 records, and a crash replays at most one batch.
- Asynchronously without waiting for broker confirmation: fastest, but a delayed retry can overwrite a newer offset with an older one.
What if the consumer has no bookmark at all?
A brand new consumer group, or one whose saved offset was deleted after aging out of retention, must pick a starting point. The configuration is auto.offset.reset.
| Setting | Starts at | Use it when |
|---|---|---|
| earliest | The oldest message still kept in the partition | A pipeline needs historical data, as in a backfill or initial load |
| latest | Only messages that arrive from now on | You only care about live events, as in an alerting service |
| none | Raises an exception and halts | A missing bookmark indicates an operational issue that requires investigation |
How do you tell a consumer is behind?
Consumer lag is the newest offset in a partition minus the consumer group's committed offset. If partition 2 is at offset 850,000 and the group has committed 820,000, the lag is 30,000 messages.
Lag holding steady near zero means consumers keep pace with incoming traffic. Lag climbing during peak traffic hours means workers are slower than incoming volume, so add consumers or optimize warehouse writes.
Engineers can also reset a group's offset using administrative tools, such as moving the bookmark back 4 hours to replay records after a bug fix.
Common beginner questions
Is the log destroyed after reading?
No. Unlike a traditional queue where consumed items are popped and deleted, Kafka retains messages for a configured retention period. Reading does not delete anything. This is what makes replay possible: a consumer can seek back to any offset within the retention window.
What happens if I never commit?
If the consumer restarts, it reprocesses everything from offset 0 (or from its last committed offset). This duplicates work, but leaves data correct if downstream writes are idempotent.
Can 2 consumers have different offsets on the same partition?
Yes, if they belong to different consumer groups. Each consumer group maintains its own independent offset tracking for every partition.
In Change Data Capture (CDC, see [CDC to raw landing](/learn/orchestration-pipelines/orch-cdc)), database replication uses Log Sequence Numbers (LSNs) as bookmarks. Kafka offsets follow the exact same principle: a sequential integer marking progress so a consumer can restart safely after failure.
Worker: A worker is counting clicks. It has read up to offset 50,000,000 and has committed that bookmark.
- read
- process
- commit 50,000,000
Broker: At 3 AM the server reboots. The worker's memory is gone, but the bookmark is safe in the broker.
- reboot
- memory lost
- bookmark kept
Worker: The worker asks for its bookmark and resumes from offset 50,000,000. It does not start over and it does not skip ahead.
- ask bookmark
- resume at 50,000,000
Worker: Because it commits after writing, the worst case is that one small batch is read twice. Nothing is lost.
- write row
- then commit
Monitor: Lag shows whether the worker keeps up. The gap between the newest offset and the bookmark is the number to watch.
- newest offset
- minus bookmark
- equals lag
What comes next
A single consumer reading from 1 partition is fine for moderate throughput. To scale out across multiple partitions, multiple consumers join together in a consumer group. That is the next lesson, on consumer groups and rebalances.
Practice
Implement next_offset(messages, start, n) on PARTITION from offset 0 taking 2 messages. Result is 2.
Going deeper: commits, crashes and replay
Offsets feel simple until the first crash. This section pins down three details that cause real incidents: what number a commit actually stores, how often you commit, and what a consumer does when it has no saved position at all.
The number you commit is the next offset, not the last one
A partition holds records numbered 0, 1, 2 and so on. If you have finished the record at offset 6, the position you save is 7: the next record to read. This off-by-one is the classic bug: commit 6 and your consumer re-reads record 6 after every restart.
Committed position means 'start here after a restart'.
| Offset | Record | Status once offset 6 is finished |
|---|---|---|
| 0 to 6 | Handled | Behind the bookmark |
| 7 | Next unread record | Committed position = 7 |
| 8 and 9 | Not yet read | Ahead of the bookmark |
Trace a crash with a commit every 5 records
Committing after every record is expensive, so consumers commit in batches. 10 records, a commit after every five handled, and a crash while the consumer is about to handle offset 7. Predict which offsets are handled twice before you read the output.
log = list(range(10)) # offsets 0..9 in 1 partition
committed = 0
processed = []
def consume(crash_at=None, batch=5):
global committed
pos, since = committed, 0
while pos < len(log):
if pos == crash_at:
return "crashed at offset %d" % pos
processed.append(pos)
pos += 1
since += 1
if since == batch: # commit the NEXT offset to read
committed, since = pos, 0
committed = pos
return "done"
print(consume(crash_at=7), "committed =", committed)
print(consume(), "committed =", committed)
print("processed order:", processed)
print("lag before restart:", len(log) - 5)Which two offsets do you expect to see twice in the processed list?
crashed at offset 7 committed = 5 done committed = 10 processed order: [0, 1, 2, 3, 4, 5, 6, 5, 6, 7, 8, 9] lag before restart: 5
Offsets 5 and 6 were handled before the crash but not yet committed, so they are handled again. The size of the replay is always 'records since the last commit'. A bigger batch means fewer commits and a larger replay; a smaller batch means more commits and a smaller replay. Neither is free.
- handle 5
- handle 6
- crash before 7
- restart at 5
- handle 5 and 6 again
What if there is no saved position?
A brand-new consumer group has never committed anything. The setting auto.offset.reset decides where it starts. It only applies when there is no valid committed position (a new group, or the saved offset no longer exists because the data aged out of retention).
| auto.offset.reset | A new group starts at | Use it when | Risk |
|---|---|---|---|
| earliest | The oldest record still retained | A pipeline must see all history, such as a new warehouse sink | A long catch-up, and a flood of old events |
| latest | Records that arrive after the consumer joins | A live alert only cares about now | Silently skips everything produced while the consumer was down for the first time |
| none | Nothing. It raises an error | You want a human to decide | The job fails until someone sets a position |
With enable.auto.commit=true a background timer saves the position of records already handed to your code. If your code is still writing them to the warehouse when it crashes, the position has moved past records that never landed. For a sink you care about, commit manually after the write succeeds.
Lag in one line
Consumer lag per partition is the log-end offset minus the committed offset. In the trace above, the end is 10 and the commit is 5, so lag is 5. Lag that stays near zero means the consumer keeps up.
Lag that grows steadily means it is slower than the producers. A sudden spike followed by recovery usually means a restart or a rebalance.
Try a new scenario
A partition holds offsets 0 to 19. A consumer commits after every 4 records and crashes just before handling offset 11. Where does the restarted consumer start, which offsets are handled twice, and what is the lag at the moment of the crash?
Work it out, then open the answer
Commits happen after offsets 3, 7 and 11 would be handled, so the saved position values are 4, 8, 12. The consumer handled 0 to 10 but only reached the commit at position 8, so the saved position is 8. Restart begins at offset 8; offsets 8, 9 and 10 are handled twice. Lag at the crash is 20 minus 8, which is 12 (the log end offset is 20).
A replayed record writes a duplicate row. Is the offset logic broken?
No. Replay after a crash is the designed behaviour of at-least-once processing. The fix belongs in the sink: an upsert or unique key on the record's business id.
Close the lesson and explain it
Without scrolling up, explain what the committed offset means and why the replay size depends on the commit interval. Then answer: if a consumer commits every 1,000 records and crashes, what is the most it can re-handle after restart?
Check yourself
Up to 1,000 records, because the last commit could be at most 1,000 handled records old. If the sink is idempotent those repeats change nothing.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.
Practice this
Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.