Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

Transformations vs actions

Easytransformationsactionslazy-evaluation

Interview question

What is the difference between transformations and actions in Spark?

How to answer

Transformations

Return a new Dataset//RDD and are lazy. Examples: select, filter, withColumn, join, groupBy, repartition.

Actions

Trigger execution and return a value to the driver or write data out. Examples: count, collect, show, take, reduce, write, foreach.

Diagram

Output
Transformations (lazy)          Actions (eager)
----------------------          ---------------
filter / map / select           count / collect
join / groupBy / window         show / take
repartition / coalesce          write.* / foreach

Code

Python
df2 = df.filter(df.status == "PAID")   # transformation
df3 = df2.withColumn("amt", df2.amount * 1.18)  # transformation

n = df3.count()                        # action -> Spark job
df3.write.mode("overwrite").parquet("/tmp/paid")  # action

Some transformations are narrow (no ), some are wide (shuffle). That affects stages, not whether something is a transformation vs action.

Interview tip

List 3 of each quickly, then mention and that excessive actions hurt performance.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill