Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Joins & Performance

What is a broadcast join?

Hardbroadcast-joinjoinsperformance

Interview question

What is a broadcast join in Spark, and when should you use it?

How to answer

The short answer

A broadcast (map-side) join copies the small side to every executor, then joins locally. It avoids shuffling the large .

Diagram

Output
Small dim ---broadcast---> Executor1 \
                           Executor2  +-- local join with local fact partitions
Large fact --no shuffle--> Executor3 /

Code

Python
from pyspark.sql.functions import broadcast

fact = spark.read.parquet("/lake/fact_orders")      # huge
dim = spark.read.parquet("/lake/dim_users")         # small

out = fact.join(broadcast(dim), "user_id", "inner")

When it helps

  • , lookup codes, small filter lists
  • Below broadcast threshold (default often 10MB, configurable via spark.sql.autoBroadcastJoinThreshold)

When it hurts / fails

  • "Small" table is actually large -> driver/executor OOM
  • Streaming unstable state sizes
  • Threshold too aggressive for skewed clusters

Interview tip

Mention can convert sort-merge joins to broadcast at runtime when size estimates warrant it, and that you still hint when you know the dimension is tiny.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill