Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

What is lazy evaluation?

Easylazy-evaluationexecutionbasics

Interview question

What is lazy evaluation in Spark, and why does it matter?

How to answer

The short answer

means Spark does not run transformations when you call them. It records a plan (lineage) and executes only when an action forces a result.

Why that helps

Spark can optimize the full pipeline before running: combine filters, prune columns, push into file readers, and avoid unnecessary .

Flow

Example

Python
df = spark.read.parquet("/data/orders")          # lazy
f = df.filter("amount > 100")                    # lazy
s = f.select("order_id", "amount")               # still lazy
print(s.count())                                 # action: job runs now
s.write.mode("overwrite").parquet("/out/orders") # action: job runs again

Interview pitfalls

  • Calling many actions (count, show, collect) recomputes from scratch unless you cache/persist or write intermediate results.
  • Errors in transformations often appear only at action time, which can surprise beginners.

One-liner

"Transformations describe work; actions trigger work."

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill