Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

What is a shuffle?

Mediumshuffleperformancewide-transformation

Interview question

What is a shuffle in Spark, and when does it happen?

How to answer

The short answer

A is the redistribution of data across so that rows that belong together (same key, same range) land on the same executor partition. It is the expensive part of .

What happens under the hood

Output
Map side (write shuffle files)     Reduce side (read + compute)
------------------------------     ----------------------------
Executor tasks hash/sort keys  --> network/disk fetch --> tasks aggregate/join
  1. 1Map tasks write shuffle blocks partitioned by key hash (or range).
  2. 2Data moves over the network (and often spills to disk).
  3. 3Reduce tasks read their assigned blocks and continue the plan.

Operations that shuffle

groupBy/agg, most joins, distinct, repartition(n), orderBy, with partitioning, cogroup, etc.

Cost drivers

  • Data volume and skew
  • Number of shuffle partitions (spark.sql.shuffle.partitions, coalescing)
  • Serializer, compression, disk spill
  • Network bandwidth

Example

Python
# Shuffle on user_id
revenue = orders.groupBy("user_id").sum("amount")

# Often avoid shuffle with broadcast if one side is small
from pyspark.sql.functions import broadcast
joined = orders.join(broadcast(dim_users), "user_id")

Interview closer

"Shuffle is necessary for correctness when keys must meet. The craft is cutting shuffled bytes and handling skew."

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill