An action creates a job. Spark cuts each job into stages at shuffle boundaries. Each stage runs one task per partition, and tasks run on executor cores. So the chain is action, job, stages, tasks.
How a real line of code maps
(df.groupBy("region").count()
.write.parquet("/out/region_counts"))groupBy and count are transformations, so nothing runs yet. The write is the action, so it starts one job. That job usually has two stages:
Stage 0: read parquet -> partial count per region -> write shuffle files
(one task per input partition)
----- shuffle -----
Stage 1: read shuffle files -> final count per region -> write parquet
(one task per shuffle partition, 200 by default)Everything that can run without moving data between machines stays in one stage. The moment rows with the same key must meet (groupBy, join, repartition), Spark needs a shuffle, and that ends the stage.
Counting tasks
If the input has 400 files of about 128 MB, stage 0 has about 400 tasks. Stage 1 has spark.sql.shuffle.partitions tasks, 200 unless you changed it or adaptive execution coalesced them. With 100 cores, stage 0 runs in about four waves of tasks.
Reading it in the Spark UI
The Jobs tab lists one row per action. Click a job to see its stages. Click a stage to see its tasks, with duration, input size, shuffle read and write, and spill. Two things to look for:
- A stage where max task time is far above the median points to skew.
- A stage marked "skipped" means Spark reused shuffle output from an earlier job, so it did not recompute it.
With adaptive query execution on, one SQL query can show extra jobs and stages, because Spark runs part of the plan, looks at real sizes, then plans the rest. Do not be surprised if the stage count is higher than what you count by hand.