Batch & Streaming · Batch Processing

What is lazy evaluation in Spark?

Easylazy evaluationCatalystactionsSpark

Interview question

What does lazy evaluation mean in Spark?

How to answer

The short answer

means Spark does not run your transformations when you call them. It records a plan (lineage / logical plan). Work starts only when an action needs a result (count, collect, write, show).

Output
df2 = df.filter(...)     # builds plan, no job yet
df3 = df2.select(...)    # still just plan
df3.write.parquet(...)   # ACTION -> Spark builds stages and runs jobs

Why it helps

  • Catalyst can optimize the whole plan (, )
  • Avoids launching useless jobs for dead-end transforms
  • Lineage supports recomputation after failure

Gotcha for freshers

Calling df.count() "to check" mid-pipeline triggers a full job. Debugging with too many actions is expensive.

Interview tip: "Transformations are lazy; actions trigger computation." Pair with the next question on transforms vs actions.