The short answer
Mitigation depends on whether skew is on join keys, aggregations, or input file sizes.
Techniques
- 1Filter / isolate hot keys: process the mega-key separately, union results.
- 2: add a random salt to hot keys on both sides (for joins) or on the group key, aggregate partial then final.
- 3 skew join optimization: enable adaptive skew join so Spark splits skewed .
- 4Broadcast the small side so the large skewed fact is not shuffle-joined.
- 5Increase parallelism cautiously (more partitions) - helps some imbalance, not true hot keys.
- 6 by better keys or two-phase aggregation (
reduceByKey-style partial aggs viagroupByalready does partial aggregation).
Salting sketch
from pyspark.sql import functions as F
salt_n = 16
hot = fact.withColumn("salt", (F.rand() * salt_n).cast("int"))
dim_exp = dim.withColumn("salt", F.explode(F.array([F.lit(i) for i in range(salt_n)])))
joined = hot.join(dim_exp, ["user_id", "salt"])Diagram
Before: key=K ######################## -> 1 task
After salt: K#0 #### K#1 #### ... K#15 #### -> many tasksInterview tip
Lead with measurement (UI + key counts), then pick the cheapest fix (broadcast / AQE) before hand-rolled salting.