Day 23Phase 05 · Spark / PySpark
Spark Fundamentals
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Explain distributed computing topology: driver, cluster manager, executors, and worker tasks.
- Distinguish between lazy transformations and eager actions that trigger Catalyst physical execution.
- Trace lineage graphs and understand how Spark recovers lost partitions without recomputing the entire pipeline.
Why One Machine Is Not Enough: Distributed Computing
Your laptop has 16 GB of RAM. ShopKart’s order history is 500 GB.
The data does not fit. Memory is only the first of 3 problems.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: horizontal scaling, driver vs executor, Catalyst DAG, transformations vs actions, narrow vs wide stages, fault tolerance lineage
Build: Distributed execution simulator: driver scheduling, narrow transformations, shuffle partitions, and worker failure recovery.
Spark coordinates distributed compute. Never run df.collect() on large data, and remember files in S3 are not partitions.
Already have a pass? Go to your account. Need a pass? See pricing.