The short answer
Network retries can create duplicates. Example:
- 1Producer sends a batch.
- 2Broker writes it successfully.
- 3ACK is lost on the network.
- 4Producer retries and sends the same batch again.
- 5Without protection, the log now has duplicates.
Idempotent producer
When enable.idempotence=true (default in modern clients), Kafka deduplicates producer retries within a producer session / epoch for a given .
Mechanism (high level):
- Producer gets a Producer ID (PID).
- Each batch has a sequence number per partition.
- Broker rejects duplicates of the same PID + sequence.
Without idempotence: retry → possible duplicate records With idempotence: retry → broker ignores duplicate sequence
What it does / does not do
- Does: prevent duplicates caused by producer retries to the same partition.
- Does not alone: give full end-to-end exactly-once across consume → transform → produce to another topic (that needs transactions / EOS streams).
- Still possible to get duplicates if your app sends the logical event twice as two different produce calls.
Related settings
Idempotence typically requires acks=all and limits on concurrent in-flight requests; modern clients configure this for you when idempotence is on.
Interview tip: "Idempotent producer = safe retries without broker-side duplicates for the same produce attempt. Exactly-once across topics needs transactions too."