Lesson 3 of 5 in this module

3/21 in track

What is a cluster?

pyspark12 min

Overview

The driver runs your script and executors process the partitions, so many machines can work on one job at once.

On this page8 sections

The picture

A ShopKart table has grown past what one laptop can hold. The script that worked yesterday now runs out of memory and dies. The company has a cluster of machines, but nobody has explained how a script uses them.

You do not rewrite everything. One program plans the work and many machines carry it out. That split is the heart of Spark.

What is the physical layout of an Apache Spark cluster?

When processing massive tables in production pipelines, a single computer runs out of memory.

Apache Spark coordinates multiple servers that pool processing power while keeping physical memory separated.

One driver, several executors
Driver / foremanExecutor AExecutor BExecutor C

The driver coordinates the job and schedules tasks. Each executor processes assigned partitions of rows in parallel.

One coordinating program acts as the driver, a cluster manager allocates servers, and executors run tasks in parallel.

The driver coordinates tasks and builds plans; executors compute partitions on worker nodes.

How does data and task separation operate across the network?

Your Python script runs exclusively on the driver node.

The driver dispatches task bytecode across the network to executor processes running on worker machines.

Apache Spark cluster architecture showing the Driver Program, a Cluster Manager, and Worker Nodes with Executors that run tasks
The official Apache Spark cluster overview: the driver program communicates with a cluster manager and dispatches tasks to executors on worker nodes.Source: Apache Software Foundation (ASF) Documentation · Apache License 2.0

Executors process assigned slices called partitions, holding intermediate records in their own memory.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
DriverThe program that plans the work and hands it out
ExecutorA worker process that runs the tasks
TaskOne piece of work on one partition
PartitionOne slice of the data
Cluster managerThe part that gives Spark its machines
StageA group of tasks that can run without moving data between machines

Why this shape

Why is distributed compute faster than scaling a single machine?

Consider calculating revenue across 50 million transactions. A single machine may take hours or run out of memory.

Split into 100 partitions, the same job can run on 24-core executors at once. In an ideal case that turns hours into minutes. The real speedup depends on the data and the job.

  1. 50,000,000 raw transactions
  2. Split into 100 partitions
  1. 20 executors compute in parallel
  2. In an ideal case, hours become minutes

Distributing the work removes the limit of one machine's memory and CPU, at the price of network and coordination costs.

Why does calling collect crash the driver node?

This design also protects the driver, if you use it carefully.

Compare asking executors for a summary count with asking them to send raw rows to the driver:

Two panels. On the left, 3 executors each calculate and send a small count number to the driver. On the right, 3 executors send millions of raw data rows to the driver along thick red arrows.
Left: count() computes locally on executors and sends one small integer. Right: collect() pulls every row into the driver process, overflowing driver memory.

On the left, each executor scans its local partition, counts matches, and returns an integer. The driver merely sums three numbers.

On the right, collect pulls every row into the driver's memory. On a large table that can exhaust the driver and fail the job.

Walk the boxes

Which components make up the core cluster architecture?

To write reliable distributed pipelines and interpret the Spark Web UI, you must distinguish between each component:

The core components of an Apache Spark cluster.

ComponentExecution roleKey architectural takeaway
DriverRuns your Python script, builds execution DAG, schedules tasksCoordinates work. Never collect large datasets here.
Cluster ManagerAllocates physical CPUs and memory across applicationsKnows which machines are free (e.g. Kubernetes, YARN).
Worker NodeA physical or virtual server in the clusterThe physical hardware hosting one or more executor processes.
ExecutorA dedicated worker process executing assigned tasksWhere partitions live and where transformations run.
PartitionAn immutable chunk of DataFrame rowsThe atomic unit of parallel execution.
TaskOne computational pass applied to 1 partitionOne stage with 200 partitions generates approximately 200 tasks.

Each component has a clear boundary, separating hardware servers from running JVM processes.

How does a DataFrame divide into partitions and tasks?

Partitions represent the atomic unit of distributed parallelism in Spark.

A DataFrame split into parallel partitions
orders p0orders p1orders p2

Each partition contains a slice of rows. Executors evaluate filters and projections on partitions independently without network movement.

Transformations like filter or select are lazy instructions stored in a lineage plan.

When you trigger an action like count or show, the driver compiles the plan into tasks and broadcasts them to executors.

A matching example

How do executors evaluate filters and count actions?

Let us inspect how this separation operates in practice. We query orders, filter for paid records, count matches, and preview 5 rows:

PythonFilter, count action, and bounded preview
df = spark.table("orders").filter("order_status = 'paid'")
print("paid count:", df.count())
result = df.limit(5)
result.show()

How many rows will df.count report, and how many rows will result.show display on your screen?

paid count: 93
order_id user_id order_status  order_total currency          created_at          updated_at
ORD-0001    U004         paid       612.07      INR 2026-08-02 16:33:28 2026-08-02 19:33:28
ORD-0003    U078         paid      1104.32      INR 2026-08-06 09:58:04 2026-08-06 11:58:04
ORD-0005    U012         paid       261.73      INR 2026-08-14 07:06:40 2026-08-14 08:06:40
ORD-0006    U036         paid       431.34      INR 2026-08-09 03:10:41 2026-08-09 08:10:41
ORD-0007    U026         paid       396.72      INR 2026-08-12 14:37:57 2026-08-12 16:37:57
(5 rows)

Executors evaluate the filter locally. The count action asks each executor for an integer, and the driver adds them up: 93 paid orders. show() then prints only 5 rows.

Why does lazy evaluation delay cluster task execution?

What happens when you define DataFrame transformations without calling an action?

PythonDefining the execution plan before triggering actions
orders = spark.table("orders")
paid = orders.filter("order_status = 'paid'")
print("still a plan until an action")
print("count =", paid.count())
result = paid.limit(5)
result.show()

Will creating the paid DataFrame trigger task execution across worker nodes?

still a plan until an action
count = 93
order_id user_id order_status  order_total currency          created_at          updated_at
ORD-0001    U004         paid       612.07      INR 2026-08-02 16:33:28 2026-08-02 19:33:28
ORD-0003    U078         paid      1104.32      INR 2026-08-06 09:58:04 2026-08-06 11:58:04
ORD-0005    U012         paid       261.73      INR 2026-08-14 07:06:40 2026-08-14 08:06:40
ORD-0006    U036         paid       431.34      INR 2026-08-09 03:10:41 2026-08-09 08:10:41
ORD-0007    U026         paid       396.72      INR 2026-08-12 14:37:57 2026-08-12 16:37:57
(5 rows)

The filter call registers a logical transformation. Physical cluster execution begins only when paid.count runs.

Watch outNever use collect() to inspect data

Beginners often call .collect() thinking it works like .show(). The method .show() prints a formatted table in the console, whereas .collect() pulls all rows from all executors across the network into a native Python list on the driver. On large datasets, this can exhaust the driver heap and fail your application with a java.lang.OutOfMemoryError.

RememberMental model: pandas versus Spark

In pandas, your entire DataFrame lives inside a single Python process on one machine; operations run immediately in local RAM. In Spark, the DataFrame is a logical pointer to data distributed across many executor processes. The driver plans, the executors compute.

How does a distributed job execute across cluster nodes?

Distributed execution lifecycle of a Spark cluster job
  1. Driver program compiles logical plan into DAG stages.

    1. Python Script
    2. Logical Plan
    3. DAG Scheduler
  2. Cluster manager allocates executor slots across worker nodes.

    1. YARN / K8s
    2. Worker Nodes
    3. JVM Executors
  3. Executors process partition tasks in parallel and return partial results.

    1. Partition Slices
    2. Task Execution
    3. Partial Counts
  4. Driver aggregates partial responses into final console output.

    1. Driver Aggregation
    2. Console Print
    3. show() Table

4 core takeaways to remember from this lesson:

  1. The driver program coordinates execution while executors process partition slices.
  2. Cluster managers allocate physical compute resources across worker nodes.
  3. Avoid collect on large tables, because it pulls all the data into driver RAM.
  4. Transformations build lazy plans while actions trigger physical cluster computation.

Common beginner questions

Does this browser editor run a real multi-machine cluster?

No. The editor runs a simulated execution environment inside your browser tab to teach the PySpark API and mental model. On a production cloud cluster, the Spark Web UI displays real physical driver and executor CPU cores, heap memory metrics, and task shuffle volumes.

How does Spark decide how many partitions a DataFrame gets?

When reading data from object storage like Amazon S3 or Google Cloud Storage, Spark creates partitions based on input file splits (typically 128 MB per partition). When reading smaller tables or executing shuffles, Spark uses configuration defaults like spark.sql.shuffle.partitions.

Why does Spark separate worker nodes from executors?

A worker node is a physical or virtual machine with hardware resources. An executor is a Java Virtual Machine (JVM) process running on that machine. A single worker node with 64 cores can host multiple smaller executors to isolate garbage collection pauses and balance load.

How do you explain the difference between a transformation and an action in an interview?

A transformation is a lazy operation that returns a new DataFrame lineage without executing computation, such as filter(), select(), or groupBy(). An action is an eager operation that triggers the compilation and execution of the DAG across the cluster to return a value to the driver or write data to external storage, such as count(), collect(), show(), or write.save().

What comes next

Now that you understand how the driver schedules tasks across executors and partitions, you are ready to manipulate data sources. In the next lesson, you will learn how Spark reads and writes diverse file formats, including Parquet, CSV, and JSON, across distributed storage systems.

Practice

In the workspace exercise, you will load the orders dataset using spark.table("orders"), filter for paid transactions, print the count, and assign a 5-row limit of the filtered DataFrame to result. Verify that your plan filters data before limiting output, keeping driver memory protected.

Going deeper: from one line of code to tasks on executors

The lesson above named the parts: driver, cluster manager, executors, partitions, tasks. This section connects them by following one real operation from your code to the machines, then lists what fails and where to look.

The job we will trace

PythonA read, a filter, a group, a write
from pyspark.sql import functions as F

orders = spark.read.parquet("s3://shop/orders/")                 # about 12.8 GB of files
paid = orders.filter("order_status = 'paid'")
by_country = paid.groupBy("country").agg(F.sum("order_total").alias("revenue"))
by_country.write.mode("overwrite").parquet("s3://shop/revenue_by_country/")

Nothing moves until an action asks for a result.

LineWhat happensDoes anything run yet?
read.parquetThe driver reads file metadata and builds a plan for readingNo data is processed
filterThe driver adds a filter step to the planNo. This is a lazy transformation
groupBy and aggThe driver adds a grouping step; rows with the same country must meetNo
write.parquetThis is an action. The driver turns the whole plan into a jobYes, now everything runs

From job to stages to tasks

  1. 1 jobcreated by the write action
  2. Stage 1read, filter, partial sum per partition
  1. shufflerows with the same country are sent to the same place
  1. Stage 2add partial sums, write files

A stage is a group of steps that can run without moving rows between machines. Reading, filtering and a per-partition partial sum only need the data already in each partition, so they share Stage 1.

Combining all the rows for one country needs rows from every partition, so Spark inserts a shuffle and starts Stage 2. The count of shuffles is a good guess for the count of stages minus one.

Each stage runs one task per partition. Spark reads files in pieces of about 128 MB by default, so 12,800 MB is 100 input partitions and 100 tasks in Stage 1. The shuffle setting spark.sql.shuffle.partitions defaults to 200, so Stage 2 starts with 200 tasks (adaptive query execution may merge small ones).

Tasks run in slots: executors times cores per executor. Predict how many rounds ('waves') of tasks a cluster with 5 executors and 4 cores each needs for each stage.

PythonPython
import math

input_mb, split_mb = 12_800, 128        # input size, and Spark's default read split size
executors, cores_each = 5, 4
slots = executors * cores_each          # tasks that can run at the same moment

stage1_tasks = math.ceil(input_mb / split_mb)
stage2_tasks = 200                      # spark.sql.shuffle.partitions default

print("task slots:", slots)
print("stage 1:", stage1_tasks, "tasks ->", math.ceil(stage1_tasks / slots), "waves")
print("stage 2:", stage2_tasks, "tasks ->", math.ceil(stage2_tasks / slots), "waves")
task slots: 20
stage 1: 100 tasks -> 5 waves
stage 2: 200 tasks -> 10 waves

Stage 1 runs in 5 waves of 20 tasks, then Stage 2 in 10 waves. Doubling executors to 10 would halve the waves, but only if the tasks are about equal in size, which is why the skew lesson matters.

Who does what while it runs

MomentDriverCluster managerExecutors
StartRuns your script, asks for resourcesFinds machines and launches executor processesStart and register with the driver
Running Stage 1Sends 100 tasks, tracks which finishedNot involved per taskEach runs tasks on its partitions and writes shuffle files to local disk
Between stagesPlans Stage 2 from the shuffle output sizesNot involvedServe their shuffle files to Stage 2 tasks
FinishRecords the result, ends the applicationReleases the machinesShut down

Two actions, 2 jobs

Each action starts a new job. A script that calls df.count() and then df.show() runs 2 jobs, and unless you cache df, both read and filter the data again from the start. Laziness is why adding a print can double a runtime.

What breaks, and where to look

The Stages page of the Spark UI answers most of these.

SymptomUsual causeWhere to look first
Driver out of memory or the job dies on collect()collect() or toPandas() pulled a big result into the driverReplace with an aggregation, limit(), or a write to storage
One task much longer than the othersSkew: 1 partition holds a hot keyStage page, task duration spread (min, median, max)
Large Spill (Disk) numbersPartitions too large for execution memoryStage page, spill columns; add partitions or memory
Executor lost, tasks retriedContainer killed (often out of memory) or a node failedExecutors tab and the cluster manager logs; Spark reruns lost tasks
Stage 2 reruns Stage 1 workAn executor died and took its shuffle files with itStage page, attempts column
Thousands of tiny tasksToo many small input files or shuffle partitionsStage page, task count versus data size
Watch outThe driver is not a worker

Your Python variables live on the driver. A function you pass to executors is serialised and sent over the network, and executors cannot see the driver's variables unless you broadcast them. If a loop on the driver touches every row, you have turned a parallel job into a single-machine job.

Try a new scenario

A script reads a 25,600 MB table, filters it, joins it to a small dimension table using a broadcast join, runs df.count() and then df.write.parquet(...). The cluster has 8 executors with 4 cores.

How many jobs, roughly how many input tasks, how many waves for the read stage, and how many shuffle stages from the join?

Work it out, then open the answer

Two actions give 2 jobs. The read splits into 25,600 / 128 = 200 tasks. There are 8 x 4 = 32 slots, so 200 / 32 is 6.25, which rounds up to 7 waves. A broadcast join sends the small table to the executors and does not shuffle the big table, so the join itself adds no extra shuffle stage. Because the first action's work is repeated by the second, caching the filtered data (if memory allows) would avoid reading it twice.

Does more executors always mean faster?

No. Speed-up stops when there are no more tasks than slots, when one task dominates (skew), when the storage or network is the limit, or when shuffle volume grows with cluster size. Measure the Stage page before you add machines.

Close the lesson and explain it

Without scrolling up, explain what makes Spark create a new stage, and why the same operations can run in a single stage when no key needs to meet another.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?