Day 26Phase 05 · Spark / PySpark

Spark Performance

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Optimize dimension joins using BroadcastHashJoin to eliminate expensive network shuffles.
  • Identify and remediate data skew bottlenecks using salt keys and adaptive query execution.
  • Decide when caching (cache, persist) improves performance versus when it causes garbage collection stalls.

Distributed Joins: Broadcast vs Sort Merge

On a single computer, joining 2 tables is straightforward because all rows reside in the same memory space.

A join cannot execute until matching keys reside on the same CPU core. Spark must coordinate data movement across workers.

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: BroadcastHashJoin (<100MB), partition pruning, Parquet predicate pushdown, when to cache vs unpersist, data skew key salting, small files coalesce

Build: Refactor a slow Spark job with 6 bottlenecks (wide scan, missing broadcast, shuffle skew, coalesce(1)) into a 95% faster pipeline.

The fastest data is data Spark never reads or moves. Use broadcast joins for small dimensions and salt skewed keys.

Already have a pass? Go to your account. Need a pass? See pricing.