Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Caching & Reliability

How do you optimize a slow PySpark job?

Hardoptimizationperformancespark-uibest-practices

Interview question

How do you approach optimizing a slow PySpark job?

How to answer

The short answer

Use a measurement-first checklist. Do not thrash random configs.

Playbook

Output
1. Reproduce & open Spark UI / SQL plan
2. Find the longest stage (usually shuffle/join/agg)
3. Check input size, row counts, partition counts, skew
4. Fix data layout & query shape
5. Then tune configs / cluster size

Concrete levers

Python
# 1) Filter early + project needed columns
df = spark.read.parquet(path).select("user_id", "amount", "dt").filter("dt = '2024-01-01'")

# 2) Prefer built-in functions over Python UDFs
# 3) Broadcast small dimensions
# 4) Handle skew (AQE skew join, salt, isolate hot keys)
# 5) Fix small files / file sizing on read and write
# 6) Cache only for multi-action reuse
# 7) Enable AQE
spark.conf.set("spark.sql.adaptive.enabled", "true")

Common root causes

  • Unexpected / Cartesian blow-up
  • stragglers
  • Reading too much (no / )
  • Python
  • Tiny files or huge gzip splits that do not split well
  • Driver OOM from collect

Interview structure

"I start from the plan and UI metrics, reduce shuffled bytes, fix skew and file layout, avoid UDFs, then scale executors if the job is legitimately large."

What "read the plan" looks like

The plan names the expensive step. The same join of a 1,000-row table to a 10-row table, on PySpark 3.5.3 with broadcast turned off, then with a broadcast hint:

Output
no broadcast:   SortMergeJoin, with an Exchange (shuffle) on both sides
with broadcast: BroadcastHashJoin, with a BroadcastExchange of the small side only

An Exchange is a shuffle: data moves across the network. Removing one from the big side is usually the largest single win, which is why "find the shuffle in the plan" comes before changing any configuration.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill