Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · UDFs & Streaming

What is exactly-once processing?

Hardexactly-oncestreamingcheckpointfault-tolerance

Interview question

What does exactly-once processing mean in Spark Structured Streaming?

How to answer

The short answer

Exactly-once means each input record affects the result once, even if the job retries after failure. No silent duplicates, no silent drops (for the guaranteed sink/source combinations).

How Spark approaches it

Output
checkpoint (offsets + state)
   + idempotent / transactional sink
   = end-to-end exactly-once (when supported)
  1. 1Checkpoint location stores offsets and state so replay starts at the right place.
  2. 2On retry, Spark recomputes a micro-batch using those offsets.
  3. 3The sink must not double-apply side effects. File sinks use commit protocols; Kafka / Delta / Iceberg have specific guarantees; arbitrary foreach may be at-least-once unless you make writes idempotent.

Modes in practice

  • At-least-once: easier; duplicates possible on retry
  • Exactly-once end-to-end: requires replayable source + checkpoint + supporting sink
  • Effectively exactly-once: idempotent keys / merge into a table (dedupe by id)
Python
(
    df.writeStream
    .format("delta")  # example of a transactional sink ecosystem
    .option("checkpointLocation", "/chk/orders")
    .outputMode("append")
    .start("/lake/silver/orders")
)

Interview tip

Be precise: Spark can provide exactly-once within its engine semantics for supported sinks; external systems need or transactions to make the business effect exactly-once.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill