Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Caching & Reliability

What is a checkpoint?

Hardcheckpointlineagefault-tolerance

Interview question

What is a checkpoint in Spark, and how does it differ from cache?

How to answer

The short answer

A checkpoint writes intermediate data to reliable storage and cuts the lineage . After checkpointing, Spark no longer recomputes from the original sources for that Dataset; it reloads from the checkpoint directory if needed.

Cache vs checkpoint

Output
cache/persist:
  - keeps data in memory/disk on executors
  - lineage retained (can recompute if lost)
  - not a durability boundary across app failures

checkpoint:
  - writes to reliable storage (HDFS/S3/local checkpoint dir)
  - lineage truncated
  - used for long lineages / iterative algorithms / streaming state recovery patterns

Example (batch)

Python
spark.sparkContext.setCheckpointDir("s3://bucket/spark-checkpoints/")

df2 = very_long_lineage_df.checkpoint()  # eager checkpoint
# or df2 = very_long_lineage_df.localCheckpoint()  # truncates but less reliable

Streaming note

Structured Streaming uses checkpoint locations for offsets and state store recovery; that is related in spirit (fault tolerance) but configured on writeStream.option("checkpointLocation", ...).

Interview tip

Use checkpoint when lineage is deep enough that recomputation or planner overhead becomes painful, or when you need a recovery boundary. Prefer writing a curated intermediate table for most .

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill