Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

overwrite vs append

Mediumwrite-modeoverwriteappendio

Interview question

What is the difference between overwrite and append write modes in PySpark?

How to answer

The short answer

Write mode controls how the sink handles existing data.

Modes

  • `append`: add new files next to existing ones. Does not remove old data.
  • `overwrite`: replace the target path (or matching in dynamic partition overwrite).
  • Also: ignore (no-op if exists), error/errorifexists (default fail).
Python
# Append new batch
df.write.mode("append").parquet("/lake/bronze/clicks")

# Replace entire dataset at path
df.write.mode("overwrite").parquet("/lake/silver/clicks_daily")

Diagram

Output
append:     [old files] + [new files]
overwrite:  delete/replace target -> [new files only]

Production caveats

  • Blind overwrite of a shared path is dangerous in concurrent jobs.
  • Append without dedupe can double-count reprocessed batches; pair with idempotent keys or merge patterns (Delta/Iceberg MERGE).
  • For partitioned tables, dynamic partition overwrite rewrites only touched partitions instead of the whole table.
Python
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
(
    df.write
    .mode("overwrite")
    .partitionBy("dt")
    .parquet("/lake/silver/events")
)

Interview tip

Explain append vs overwrite, then mention lakehouse table formats for safer transactional writes.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill