Lazy evaluation means Spark does not run your transformations when you call them. It records a plan (lineage / logical plan). Work starts only when an action needs a result (count, collect, write, show).
df2 = df.filter(...) # builds plan, no job yet df3 = df2.select(...) # still just plan df3.write.parquet(...) # ACTION -> Spark builds stages and runs jobs
Why it helps
- Catalyst can optimize the whole plan (predicate pushdown, column pruning)
- Avoids launching useless jobs for dead-end transforms
- Lineage supports recomputation after failure
Gotcha for freshers
Calling df.count() "to check" mid-pipeline triggers a full job. Debugging with too many actions is expensive.
Interview tip: "Transformations are lazy; actions trigger computation." Pair with the next question on transforms vs actions.