Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Idempotent producer

Kafka · Producers & Consumers

Idempotent producer

Mediumkafka-13
idempotent producerduplicatesretrieseos

Question

What is an idempotent producer in Kafka, and what problem does it solve?

Solution

Network retries can create duplicates. Example:

1. Producer sends a batch. 2. Broker writes it successfully. 3. ACK is lost on the network. 4. Producer retries and sends the same batch again. 5. Without protection, the log now has duplicates.

Idempotent producer

When enable.idempotence=true (default in modern clients), Kafka deduplicates producer retries within a producer session / epoch for a given partition.

Mechanism (high level):

  • Producer gets a Producer ID (PID).
  • Each batch has a sequence number per partition.
  • Broker rejects duplicates of the same PID + sequence.
Without idempotence:  retry → possible duplicate records
With idempotence:     retry → broker ignores duplicate sequence

What it does / does not do

  • Does: prevent duplicates caused by producer retries to the same partition.
  • Does not alone: give full end-to-end exactly-once across consume → transform → produce to another topic (that needs transactions / EOS streams).
  • Still possible to get duplicates if your app sends the logical event twice as two different produce calls.

Related settings

Idempotence typically requires acks=all and limits on concurrent in-flight requests; modern clients configure this for you when idempotence is on.

Interview tip: "Idempotent producer = safe retries without broker-side duplicates for the same produce attempt. Exactly-once across topics needs transactions too."

PreviousNext