Transformations vs actions

Transformations

Return a new Dataset/DataFrame/RDD and are lazy. Examples: select, filter, withColumn, join, groupBy, repartition.

Actions

Trigger execution and return a value to the driver or write data out. Examples: count, collect, show, take, reduce, write, foreach.

Diagram

Transformations (lazy)          Actions (eager)
----------------------          ---------------
filter / map / select           count / collect
join / groupBy / window         show / take
repartition / coalesce          write.* / foreach

Code

df2 = df.filter(df.status == "PAID")   # transformation
df3 = df2.withColumn("amt", df2.amount * 1.18)  # transformation

n = df3.count()                        # action -> Spark job
df3.write.mode("overwrite").parquet("/tmp/paid")  # action

Narrow vs wide (related)

Some transformations are narrow (no shuffle), some are wide (shuffle). That affects stages, not whether something is a transformation vs action.

Interview tip

List 3 of each quickly, then mention lazy evaluation and that excessive actions hurt performance.