Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

Narrow vs wide transformations

Mediumnarrowwideshuffletransformations

Interview question

Explain narrow vs wide transformations in Spark, with examples.

How to answer

Narrow transformation

Each input contributes to at most one output partition. No of data across the network. Examples: map, filter, flatMap, withColumn, select, union (often).

Wide transformation

Input partitions contribute to many output partitions. Requires a shuffle (exchange). Examples: groupByKey/groupBy.agg, reduceByKey, join (non-broadcast), distinct, repartition, orderBy/sort.

Code

Python
# Narrow: stays in same stage when possible
df2 = df.filter("amount > 0").withColumn("tax", df.amount * 0.1)

# Wide: introduces a shuffle
by_user = df2.groupBy("user_id").sum("amount")

Why that matters

  • Narrow ops are cheaper and can be pipelined.
  • Wide ops create new stages, more disk/network I/O, and are common slow-job culprits.
  • Prefer reducing data before wide ops (filter early).

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill