Day 23Phase 05 · Spark / PySpark

Spark Fundamentals

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Explain distributed computing topology: driver, cluster manager, executors, and worker tasks.
  • Distinguish between lazy transformations and eager actions that trigger Catalyst physical execution.
  • Trace lineage graphs and understand how Spark recovers lost partitions without recomputing the entire pipeline.

Why One Machine Is Not Enough: Distributed Computing

Your laptop has 16 GB of RAM. ShopKart’s order history is 500 GB.

The data does not fit. Memory is only the first of 3 problems.

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: horizontal scaling, driver vs executor, Catalyst DAG, transformations vs actions, narrow vs wide stages, fault tolerance lineage

Build: Distributed execution simulator: driver scheduling, narrow transformations, shuffle partitions, and worker failure recovery.

Spark coordinates distributed compute. Never run df.collect() on large data, and remember files in S3 are not partitions.

Already have a pass? Go to your account. Need a pass? See pricing.