Day 25Phase 05 · Spark / PySpark
Spark Transformations
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Distinguish narrow transformations (filter, select) from wide transformations (groupBy, join) that force shuffles.
- Control partition counts and file sizes using repartition() versus coalesce().
- Prevent single-core executor crashes caused by premature coalesce(1) calls.
What is a Partition Physically?
In distributed computing, a large dataset is not stored as one giant contiguous block in memory. Spark splits the dataset into smaller chunks called partitions.
Each partition is assigned to a single thread running on a single CPU core. If your cluster has 16 total CPU cores and your DataFrame has 16 partitions, all 16 partitions are processed simultaneously in one parallel wave.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: partition boundaries, narrow vs wide transformations, shuffle write & fetch, repartition vs coalesce, coalesce(1) single-core hazard, partition sizing
Build: Multi-stage pipeline: narrow filtering → wide shuffle aggregation → hash repartitioning by customer key → coalesce(2) lake write.
Never use coalesce(1) before writing large datasets: it collapses upstream execution to a single core and crashes workers.
Already have a pass? Go to your account. Need a pass? See pricing.