True exactly-once execution is not realistic in distributed batch systems, because retries mean a task can run more than once. What you can have is effectively-once: the code may run twice, but the result is the same as if it ran once. You get it by making writes idempotent.
Why code runs twice
A task times out, the scheduler retries it, and the first attempt turns out to have finished after all. Or a worker dies after writing but before reporting success. The system cannot distinguish "did not run" from "ran but I did not hear back", so it must retry and accept duplicates in execution.
Make the effect idempotent
- Overwrite a partition for the run date. Running it twice replaces the same data with the same data.
- MERGE by business key. Running the same input twice updates the same rows to the same values.
- Deduplicate by a unique id (an event id or a hash of the row) before or after writing.
- Write to a path or table that includes the run id, and publish it atomically. A retry writes to the same place and replaces it.
Bookkeeping
For jobs that cannot be made idempotent by data alone, keep a table of processed batches (batch id, status, time). Check before starting. Record success in the same transaction as the data write if possible. If the data write and the bookkeeping write are in different systems, decide the order carefully and make the first one safe to repeat.
External side effects
Sending an email, posting to an API, or charging a card cannot be rolled back by overwriting a partition. Use an idempotency key: a unique id for each logical action (order-123-confirmation) that you send with the request, so the receiving system recognises a repeat and ignores it. If the system does not support keys, store "already sent" in your own table and check it first.
Streaming link
Kafka and Flink advertise exactly-once. Under the hood that is the same idea (transactions plus idempotent writes within a bounded system). The guarantee ends at the boundary of what they control, so a write to an external database still needs idempotency.
Answer shape
Say the phrase "effectively-once", explain that retries are normal, and give two examples of idempotent writes, plus the idempotency key for side effects.