Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. High watermark and log end offset

Kafka · Internals

High watermark and log end offset

Mediumkafka-35
high-watermarklog-end-offsetreplicationconsistency

Question

What are the high watermark and the log end offset?

Solution

The Log End Offset (LEO) is the offset of the next record to be written to a specific partition replica log. The High Watermark (HW) is the highest offset that has been copied to all replicas in the in-sync replica (ISR) set. Consumers are only permitted to read messages up to the high watermark, which prevents applications from reading uncommitted records that could disappear if the leader broker crashes.

Tracking offsets across replicas

Every partition replica on every broker tracks its own LEO independently. When a producer sends records to the partition leader, the leader appends them to its local disk and increments its local LEO. Follower brokers periodically send fetch requests to the leader to pull these new records, and the leader monitors each follower progress.

Broker 1 (Leader):   [0][1][2][3][4][5][6][7] -> LEO = 8
Broker 2 (Follower): [0][1][2][3][4][5]       -> LEO = 6
Broker 3 (Follower): [0][1][2][3][4][5]       -> LEO = 6

High Watermark (HW) = offset 6 (visible to consumers)
Offsets 6 and 7 are uncommitted and invisible.

A brief look at how the replication boundary moves:

  • The leader calculates the high watermark as the minimum LEO across all brokers in the current ISR.
  • As followers issue subsequent fetch requests acknowledging receipt of offsets 6 and 7, the leader advances the high watermark to 8.
  • The updated high watermark is returned to followers in their next fetch response, allowing them to advance their own watermarks.

Protection against dirty reads and leader failover

The high watermark acts as a barrier separating committed records from uncommitted in-flight writes. If consumers could read ahead to the leader LEO at offset 7, and the leader broker suddenly suffered hardware failure before followers fetched those messages, a follower elected as the new leader would lack offsets 6 and 7.

Downstream systems would have already acted on uncommitted records that no longer exist in the cluster. Restricting consumer fetch requests strictly up to the high watermark guarantees that every message visible to downstream readers has survived replication across the quorum and will persist through leader elections.

PreviousNext