Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Joins & Performance

How do you fix data skew?

Hardskewsaltingaqeperformance

Interview question

How do you fix or mitigate data skew in PySpark jobs?

How to answer

The short answer

Mitigation depends on whether skew is on join keys, aggregations, or input file sizes.

Techniques

  1. 1Filter / isolate hot keys: process the mega-key separately, union results.
  2. 2: add a random salt to hot keys on both sides (for joins) or on the group key, aggregate partial then final.
  3. 3 skew join optimization: enable adaptive skew join so Spark splits skewed .
  4. 4Broadcast the small side so the large skewed fact is not shuffle-joined.
  5. 5Increase parallelism cautiously (more partitions) - helps some imbalance, not true hot keys.
  6. 6 by better keys or two-phase aggregation (reduceByKey-style partial aggs via groupBy already does partial aggregation).

Salting sketch

Python
from pyspark.sql import functions as F

salt_n = 16
hot = fact.withColumn("salt", (F.rand() * salt_n).cast("int"))
dim_exp = dim.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(salt_n)])))

joined = hot.join(dim_exp, ["user_id", "salt"])

Diagram

Output
Before: key=K ########################  -> 1 task
After salt: K#0 ####  K#1 #### ... K#15 #### -> many tasks

Interview tip

Lead with measurement (UI + key counts), then pick the cheapest fix (broadcast / AQE) before hand-rolled salting.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill