Overview
The driver runs your script and executors process the partitions, so many machines can work on one job at once.
On this page8 sections
The picture
A ShopKart table has grown past what one laptop can hold. The script that worked yesterday now runs out of memory and dies. The company has a cluster of machines, but nobody has explained how a script uses them.
You do not rewrite everything. One program plans the work and many machines carry it out. That split is the heart of Spark.
What is the physical layout of an Apache Spark cluster?
When processing massive tables in production pipelines, a single computer runs out of memory.
Apache Spark coordinates multiple servers that pool processing power while keeping physical memory separated.
The driver coordinates the job and schedules tasks. Each executor processes assigned partitions of rows in parallel.
One coordinating program acts as the driver, a cluster manager allocates servers, and executors run tasks in parallel.
The driver coordinates tasks and builds plans; executors compute partitions on worker nodes.
How does data and task separation operate across the network?
Your Python script runs exclusively on the driver node.
The driver dispatches task bytecode across the network to executor processes running on worker machines.

Executors process assigned slices called partitions, holding intermediate records in their own memory.
Words you will see in this lesson
Six words. The rest of the lesson uses only these.
| Word | What it means |
|---|---|
| Driver | The program that plans the work and hands it out |
| Executor | A worker process that runs the tasks |
| Task | One piece of work on one partition |
| Partition | One slice of the data |
| Cluster manager | The part that gives Spark its machines |
| Stage | A group of tasks that can run without moving data between machines |
Why this shape
Why is distributed compute faster than scaling a single machine?
Consider calculating revenue across 50 million transactions. A single machine may take hours or run out of memory.
Split into 100 partitions, the same job can run on 24-core executors at once. In an ideal case that turns hours into minutes. The real speedup depends on the data and the job.
- 50,000,000 raw transactions
- Split into 100 partitions
- 20 executors compute in parallel
- In an ideal case, hours become minutes
Distributing the work removes the limit of one machine's memory and CPU, at the price of network and coordination costs.
Why does calling collect crash the driver node?
This design also protects the driver, if you use it carefully.
Compare asking executors for a summary count with asking them to send raw rows to the driver:
On the left, each executor scans its local partition, counts matches, and returns an integer. The driver merely sums three numbers.
On the right, collect pulls every row into the driver's memory. On a large table that can exhaust the driver and fail the job.
Walk the boxes
Which components make up the core cluster architecture?
To write reliable distributed pipelines and interpret the Spark Web UI, you must distinguish between each component:
The core components of an Apache Spark cluster.
| Component | Execution role | Key architectural takeaway |
|---|---|---|
| Driver | Runs your Python script, builds execution DAG, schedules tasks | Coordinates work. Never collect large datasets here. |
| Cluster Manager | Allocates physical CPUs and memory across applications | Knows which machines are free (e.g. Kubernetes, YARN). |
| Worker Node | A physical or virtual server in the cluster | The physical hardware hosting one or more executor processes. |
| Executor | A dedicated worker process executing assigned tasks | Where partitions live and where transformations run. |
| Partition | An immutable chunk of DataFrame rows | The atomic unit of parallel execution. |
| Task | One computational pass applied to 1 partition | One stage with 200 partitions generates approximately 200 tasks. |
Each component has a clear boundary, separating hardware servers from running JVM processes.
How does a DataFrame divide into partitions and tasks?
Partitions represent the atomic unit of distributed parallelism in Spark.
Each partition contains a slice of rows. Executors evaluate filters and projections on partitions independently without network movement.
Transformations like filter or select are lazy instructions stored in a lineage plan.
When you trigger an action like count or show, the driver compiles the plan into tasks and broadcasts them to executors.
A matching example
How do executors evaluate filters and count actions?
Let us inspect how this separation operates in practice. We query orders, filter for paid records, count matches, and preview 5 rows:
df = spark.table("orders").filter("order_status = 'paid'")
print("paid count:", df.count())
result = df.limit(5)
result.show()How many rows will df.count report, and how many rows will result.show display on your screen?
paid count: 93 order_id user_id order_status order_total currency created_at updated_at ORD-0001 U004 paid 612.07 INR 2026-08-02 16:33:28 2026-08-02 19:33:28 ORD-0003 U078 paid 1104.32 INR 2026-08-06 09:58:04 2026-08-06 11:58:04 ORD-0005 U012 paid 261.73 INR 2026-08-14 07:06:40 2026-08-14 08:06:40 ORD-0006 U036 paid 431.34 INR 2026-08-09 03:10:41 2026-08-09 08:10:41 ORD-0007 U026 paid 396.72 INR 2026-08-12 14:37:57 2026-08-12 16:37:57 (5 rows)
Executors evaluate the filter locally. The count action asks each executor for an integer, and the driver adds them up: 93 paid orders. show() then prints only 5 rows.
Why does lazy evaluation delay cluster task execution?
What happens when you define DataFrame transformations without calling an action?
orders = spark.table("orders")
paid = orders.filter("order_status = 'paid'")
print("still a plan until an action")
print("count =", paid.count())
result = paid.limit(5)
result.show()Will creating the paid DataFrame trigger task execution across worker nodes?
still a plan until an action count = 93 order_id user_id order_status order_total currency created_at updated_at ORD-0001 U004 paid 612.07 INR 2026-08-02 16:33:28 2026-08-02 19:33:28 ORD-0003 U078 paid 1104.32 INR 2026-08-06 09:58:04 2026-08-06 11:58:04 ORD-0005 U012 paid 261.73 INR 2026-08-14 07:06:40 2026-08-14 08:06:40 ORD-0006 U036 paid 431.34 INR 2026-08-09 03:10:41 2026-08-09 08:10:41 ORD-0007 U026 paid 396.72 INR 2026-08-12 14:37:57 2026-08-12 16:37:57 (5 rows)
The filter call registers a logical transformation. Physical cluster execution begins only when paid.count runs.
Beginners often call .collect() thinking it works like .show(). The method .show() prints a formatted table in the console, whereas .collect() pulls all rows from all executors across the network into a native Python list on the driver. On large datasets, this can exhaust the driver heap and fail your application with a java.lang.OutOfMemoryError.
In pandas, your entire DataFrame lives inside a single Python process on one machine; operations run immediately in local RAM. In Spark, the DataFrame is a logical pointer to data distributed across many executor processes. The driver plans, the executors compute.
How does a distributed job execute across cluster nodes?
Driver program compiles logical plan into DAG stages.
- Python Script
- Logical Plan
- DAG Scheduler
Cluster manager allocates executor slots across worker nodes.
- YARN / K8s
- Worker Nodes
- JVM Executors
Executors process partition tasks in parallel and return partial results.
- Partition Slices
- Task Execution
- Partial Counts
Driver aggregates partial responses into final console output.
- Driver Aggregation
- Console Print
- show() Table
4 core takeaways to remember from this lesson:
- The driver program coordinates execution while executors process partition slices.
- Cluster managers allocate physical compute resources across worker nodes.
- Avoid collect on large tables, because it pulls all the data into driver RAM.
- Transformations build lazy plans while actions trigger physical cluster computation.
Common beginner questions
Does this browser editor run a real multi-machine cluster?
No. The editor runs a simulated execution environment inside your browser tab to teach the PySpark API and mental model. On a production cloud cluster, the Spark Web UI displays real physical driver and executor CPU cores, heap memory metrics, and task shuffle volumes.
How does Spark decide how many partitions a DataFrame gets?
When reading data from object storage like Amazon S3 or Google Cloud Storage, Spark creates partitions based on input file splits (typically 128 MB per partition). When reading smaller tables or executing shuffles, Spark uses configuration defaults like spark.sql.shuffle.partitions.
Why does Spark separate worker nodes from executors?
A worker node is a physical or virtual machine with hardware resources. An executor is a Java Virtual Machine (JVM) process running on that machine. A single worker node with 64 cores can host multiple smaller executors to isolate garbage collection pauses and balance load.
How do you explain the difference between a transformation and an action in an interview?
A transformation is a lazy operation that returns a new DataFrame lineage without executing computation, such as filter(), select(), or groupBy(). An action is an eager operation that triggers the compilation and execution of the DAG across the cluster to return a value to the driver or write data to external storage, such as count(), collect(), show(), or write.save().
What comes next
Now that you understand how the driver schedules tasks across executors and partitions, you are ready to manipulate data sources. In the next lesson, you will learn how Spark reads and writes diverse file formats, including Parquet, CSV, and JSON, across distributed storage systems.
Practice
In the workspace exercise, you will load the orders dataset using spark.table("orders"), filter for paid transactions, print the count, and assign a 5-row limit of the filtered DataFrame to result. Verify that your plan filters data before limiting output, keeping driver memory protected.
Going deeper: from one line of code to tasks on executors
The lesson above named the parts: driver, cluster manager, executors, partitions, tasks. This section connects them by following one real operation from your code to the machines, then lists what fails and where to look.
The job we will trace
from pyspark.sql import functions as F
orders = spark.read.parquet("s3://shop/orders/") # about 12.8 GB of files
paid = orders.filter("order_status = 'paid'")
by_country = paid.groupBy("country").agg(F.sum("order_total").alias("revenue"))
by_country.write.mode("overwrite").parquet("s3://shop/revenue_by_country/")Nothing moves until an action asks for a result.
| Line | What happens | Does anything run yet? |
|---|---|---|
| read.parquet | The driver reads file metadata and builds a plan for reading | No data is processed |
| filter | The driver adds a filter step to the plan | No. This is a lazy transformation |
| groupBy and agg | The driver adds a grouping step; rows with the same country must meet | No |
| write.parquet | This is an action. The driver turns the whole plan into a job | Yes, now everything runs |
From job to stages to tasks
- 1 jobcreated by the write action
- Stage 1read, filter, partial sum per partition
- shufflerows with the same country are sent to the same place
- Stage 2add partial sums, write files
A stage is a group of steps that can run without moving rows between machines. Reading, filtering and a per-partition partial sum only need the data already in each partition, so they share Stage 1.
Combining all the rows for one country needs rows from every partition, so Spark inserts a shuffle and starts Stage 2. The count of shuffles is a good guess for the count of stages minus one.
Each stage runs one task per partition. Spark reads files in pieces of about 128 MB by default, so 12,800 MB is 100 input partitions and 100 tasks in Stage 1. The shuffle setting spark.sql.shuffle.partitions defaults to 200, so Stage 2 starts with 200 tasks (adaptive query execution may merge small ones).
Tasks run in slots: executors times cores per executor. Predict how many rounds ('waves') of tasks a cluster with 5 executors and 4 cores each needs for each stage.
import math
input_mb, split_mb = 12_800, 128 # input size, and Spark's default read split size
executors, cores_each = 5, 4
slots = executors * cores_each # tasks that can run at the same moment
stage1_tasks = math.ceil(input_mb / split_mb)
stage2_tasks = 200 # spark.sql.shuffle.partitions default
print("task slots:", slots)
print("stage 1:", stage1_tasks, "tasks ->", math.ceil(stage1_tasks / slots), "waves")
print("stage 2:", stage2_tasks, "tasks ->", math.ceil(stage2_tasks / slots), "waves")task slots: 20 stage 1: 100 tasks -> 5 waves stage 2: 200 tasks -> 10 waves
Stage 1 runs in 5 waves of 20 tasks, then Stage 2 in 10 waves. Doubling executors to 10 would halve the waves, but only if the tasks are about equal in size, which is why the skew lesson matters.
Who does what while it runs
| Moment | Driver | Cluster manager | Executors |
|---|---|---|---|
| Start | Runs your script, asks for resources | Finds machines and launches executor processes | Start and register with the driver |
| Running Stage 1 | Sends 100 tasks, tracks which finished | Not involved per task | Each runs tasks on its partitions and writes shuffle files to local disk |
| Between stages | Plans Stage 2 from the shuffle output sizes | Not involved | Serve their shuffle files to Stage 2 tasks |
| Finish | Records the result, ends the application | Releases the machines | Shut down |
Two actions, 2 jobs
Each action starts a new job. A script that calls df.count() and then df.show() runs 2 jobs, and unless you cache df, both read and filter the data again from the start. Laziness is why adding a print can double a runtime.
What breaks, and where to look
The Stages page of the Spark UI answers most of these.
| Symptom | Usual cause | Where to look first |
|---|---|---|
| Driver out of memory or the job dies on collect() | collect() or toPandas() pulled a big result into the driver | Replace with an aggregation, limit(), or a write to storage |
| One task much longer than the others | Skew: 1 partition holds a hot key | Stage page, task duration spread (min, median, max) |
| Large Spill (Disk) numbers | Partitions too large for execution memory | Stage page, spill columns; add partitions or memory |
| Executor lost, tasks retried | Container killed (often out of memory) or a node failed | Executors tab and the cluster manager logs; Spark reruns lost tasks |
| Stage 2 reruns Stage 1 work | An executor died and took its shuffle files with it | Stage page, attempts column |
| Thousands of tiny tasks | Too many small input files or shuffle partitions | Stage page, task count versus data size |
Your Python variables live on the driver. A function you pass to executors is serialised and sent over the network, and executors cannot see the driver's variables unless you broadcast them. If a loop on the driver touches every row, you have turned a parallel job into a single-machine job.
Try a new scenario
A script reads a 25,600 MB table, filters it, joins it to a small dimension table using a broadcast join, runs df.count() and then df.write.parquet(...). The cluster has 8 executors with 4 cores.
How many jobs, roughly how many input tasks, how many waves for the read stage, and how many shuffle stages from the join?
Work it out, then open the answer
Two actions give 2 jobs. The read splits into 25,600 / 128 = 200 tasks. There are 8 x 4 = 32 slots, so 200 / 32 is 6.25, which rounds up to 7 waves. A broadcast join sends the small table to the executors and does not shuffle the big table, so the join itself adds no extra shuffle stage. Because the first action's work is repeated by the second, caching the filtered data (if memory allows) would avoid reading it twice.
Does more executors always mean faster?
No. Speed-up stops when there are no more tasks than slots, when one task dominates (skew), when the storage or network is the limit, or when shuffle volume grows with cluster size. Measure the Stage page before you add machines.
Close the lesson and explain it
Without scrolling up, explain what makes Spark create a new stage, and why the same operations can run in a single stage when no key needs to meet another.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.