Lesson 1 of 3 in this module

6/21 in track

PySpark architecture & mental model

pyspark10 min

Overview

The driver builds the plan and sends work to executors. Executors process partitions. Transformations describe work first, while an action such as count or write starts the job.

On this page8 sections

The picture

A ShopKart engineer writes df.filter(...).groupBy(...).agg(...) in PySpark. It reads like pandas, but when they call show() nothing happens for a while, and then everything happens at once.

The reason is that the code you write builds a plan on the driver, and the work only runs on executors when you ask for a result.

How is an Apache Spark cluster structured?

One driver, many executors
Driver / foremanDriverExecutor AExecutor BExecutor CPartition 1Partition 2Partition 3

The driver sends tasks. Executors process data partitions in parallel.

The driver holds the SparkSession, the plan, the task scheduler and small final results. It asks a cluster manager for executors. The manager starts executors on two worker machines, each running two tasks on partitions, and the driver sends tasks straight to the executors
Three steps: the driver asks the cluster manager for executors, the manager starts them on worker machines, and the driver then sends tasks to them directly and gets small results back.

Apache Spark is a distributed computing engine for processing large datasets across many machines.

Unlike pandas, which runs on a single computer and holds all data in memory, Spark splits data into partitions and processes them in parallel across a cluster of machines.

  1. Driver Application
  2. DAG Scheduler
  3. Worker Executors
  4. Partition Tasks

Spark has 2 key components. The driver is the coordinator: it runs your Python script, builds a plan of what work to do, and schedules tasks.

Executors are the workers: they receive tasks from the driver, process partitions of data, and return results. You can think of the driver as a foreman giving instructions and executors as crews doing the physical work.

Spark uses lazy evaluation: when you write commands like filter() or select(), nothing actually happens yet. Spark just records the instructions.

The real work begins only when you trigger an action like count(), show(), or write(). This lets Spark optimize the entire sequence of operations before executing them.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
DriverThe program that plans the work and hands it out
ExecutorA worker process that runs the tasks
PartitionOne slice of the data
RDDSpark's older, low-level way to hold data
LazyNothing runs until an action asks for a result
ActionA call such as count() that makes Spark run the plan

Why this shape

Why does distributed architecture matter for large datasets?

Pandas works well for datasets that fit in your laptop's memory, typically up to a few gigabytes.

When your data grows to hundreds of gigabytes or terabytes, a single machine cannot hold it all. Spark distributes data and processing across dozens or hundreds of machines, making it possible to process datasets that are too large for pandas.

The driver is one small process. The executors are many, and they hold the data.

Architecture ComponentMemory ProfileOperational Responsibility
Driver ProcessSingle JVM process with modest RAMRuns application script, coordinates DAG
Worker ExecutorsDistributed across cluster worker nodesProcesses partition data slices in parallel

Understanding the driver-executor architecture is essential because common mistakes stem from not understanding which component does what.

Calling collect() on 1 billion rows is a prime example. The driver is small. If you try to pull all the data onto the driver, it can run out of memory and the job can fail.

Walk the boxes

What distinct roles do the driver and executors play?

In this editor, we simulate Spark's API using pandas underneath. The architecture and performance guidance describes real Apache Spark. You practice the verbs and thinking patterns here; in production, these run on actual clusters.

The driver is a single process that runs your application code. It creates a SparkSession, builds query plans, and coordinates the cluster. Think of it as the office that sends instructions. It should not hold the entire dataset.

  • Spark Cluster HierarchyDriver and worker node hierarchy in a Spark cluster
    • ├── Driver Process (Coordinator)
    • │ ├── SparkSession
    • │ └── DAG Scheduler
    • └── Worker Compute Nodes
    • ├── Executor 1 (Tasks on Partitions)
    • └── Executor 2 (Tasks on Partitions)

Executors are long-running worker processes on cluster machines. Each executor runs tasks and stores cached partition data.

A task typically handles 1 partition, so more executors and partitions mean more parallelism across your cluster.

RememberIf you know Pandas, here is the translation

In pandas, df = pd.read_csv(...) loads everything into one machine's RAM. In Spark, df = spark.read.parquet(...) creates a plan to read from distributed storage. The data stays distributed across executors until you explicitly collect it.

Why are DataFrames preferred over lower-level RDDs?

Spark has two APIs for working with data. RDDs (Resilient Distributed Datasets) are the older, lower-level API: collections of Python objects without column structure.

DataFrames are the newer, recommended API: tables with named columns and types, similar to relational tables.

  1. Your DataFrame code
  2. Spark's plan
  3. A cheaper plan chosen
  4. Fast code generated

DataFrames are almost always the right choice. They give Spark's optimizer (Catalyst) enough information to rearrange operations, skip unnecessary columns, and generate efficient code.

RDDs bypass these optimizations and force more work through Python serialization overhead.

A matching example

How does code define an RDD versus a DataFrame?

Run the example below in this tab. Read the input, follow the code, then check the output matches what you expect.

PythonRDD vs DataFrame: DataFrames have structure the optimizer can use
# RDD: lower-level, no column structure
rdd = spark.sparkContext.parallelize([("A", 10), ("B", 20)])

# DataFrame: named columns, optimizer-visible
df = spark.createDataFrame(rdd, ["zone", "parcel_count"])
df.printSchema()
df.show()

Notice how the DataFrame prints a clean schema with column names and data types, whereas the RDD is an opaque collection of raw Python tuples.

How do lazy transformations differ from eager actions?

Transformations like select, filter, withColumn, and groupBy add steps to a plan but do not execute anything. They return a new DataFrame that describes what should happen.

You can chain many transformations together without moving any bytes over the cluster network.

Transformations build plans. Actions demand results. Lazy evaluation means nothing runs until an action.

OperationCategoryWhat happens
select(...)TransformationAdds a projection to the plan. No data moves.
filter(...)TransformationAdds a row predicate to the plan. No data moves.
count()ActionTriggers execution. Returns one number to the driver.
show(20)ActionTriggers execution. Prints up to 20 rows.
collect()ActionTriggers execution. Returns ALL rows to the driver.
write.parquet(...)ActionTriggers execution. Writes distributed output files.

Actions like count, show, collect, and write trigger execution. Spark analyzes the accumulated plan, optimizes it, and schedules tasks on the executors. Only then does data actually move and get processed.

PythonA lazy plan followed by two actions
paid = (
    spark.table("orders")
    .filter("order_status = 'paid'")
    .select("order_id", "order_total")
)

print("No execution yet: paid is just a plan")
print("Paid order count:", paid.count())  # action triggers execution
paid.show(5)                              # another action
Watch outAssigning is not materializing

Writing paid = df.filter(...) creates a plan object, not a copy of filtered data. No memory is allocated, no data is read, and no CPU is used until an action runs.

How does Spark organize work using a DAG?

Spark represents the computation plan as a directed acyclic graph (DAG).

Narrow operations like filter and select can run on each partition independently without network data transfer.

Wide operations like groupBy and join require a shuffle: moving data between executors so that matching keys end up together on the same node.

From scan to answer
Scan ordersFilter paidProject columnsShuffle by statusAggregateReturn summary

Narrow operations share a stage. The shuffle creates a new stage.

Narrow versus wide transformations in DAG execution stages

Transformation TypeNetwork TransferStage Boundary
Narrow (filter, select)None: executes inside partitionPipelined in current stage
Wide (groupBy, join)High: shuffles rows across nodesCreates new stage boundary
Transformations only write the plan: read parquet, filter amount over 100, groupBy and sum. The show action starts one job. Spark cuts it at the shuffle into stage 1, four tasks reading and filtering partitions, and stage 2, three tasks adding up the partial sums per country
Nothing runs until the action. Then Spark cuts the job at the shuffle: stage 1 works on every partition at once, and stage 2 starts only after stage 1 has finished.

Shuffles are expensive because they send data across the network. They create stage boundaries in the DAG. Understanding where shuffles happen helps you write efficient Spark code.

Why must you keep bulk data off the driver node?

Calling collect() converts every distributed row into Python objects on the driver node.

If the result is larger than the driver's memory, the application can fail with an out-of-memory error.

Use show() for inspection, count() for row counts, and distributed writes for real outputs.

PythonSafe inspection patterns that avoid driver overload
orders = spark.table("orders")

# Safe: only a bounded sample reaches the driver
orders.select("order_id", "order_status").show(20, truncate=False)

# Safe: executors aggregate, driver receives one number
print("rows:", orders.count())

# Dangerous on large data:
# all_orders = orders.collect()  # can crash the driver
count() makes each of three executors send one number to the driver, which adds them up to 12,000. collect() makes every executor send all of its rows, so the driver has to hold 12,000 rows
Same executors, same data. What changes is how much the driver is asked to hold.
  • Use DataFrame expressions over raw Python logic.
  • Treat actions as execution (and cost) boundaries.
  • Assume the driver is small. Only send bounded results to it.
  • Read physical plans to find shuffles and understand cost.
Watch outlimit does not make every plan cheap

A limit bounds the returned rows, but upstream sorts, aggregations, or selective filters may still process the full dataset. Inspect the plan before assuming a limit makes the query fast.

How do you safely summarize groups without collecting rows?

The exercise below asks you to replace collect() with something safer: groupBy("order_status").agg(F.sum("order_total").alias("total_paid_amount")).

Read it the same way you would read a sentence. groupBy("order_status") gathers rows into buckets, one bucket per distinct status value, such as paid, pending, or cancelled.

  1. orders Table
  2. groupBy('order_status')
  3. agg(F.sum('order_total'))
  4. 1 Row Per Status Bucket

Then agg(...) computes one summary number per bucket instead of handing back every row.

F.sum("order_total") is the number being computed, the total order value in that bucket. The alias call gives that computed column a readable name instead of a long default.

This matters because the summary has only one small row per group, no matter how many millions of rows went into computing it. agg() itself is a transformation: it describes the summary, and an action such as show() or collect() brings those few rows back.

That is the opposite of collect() on the raw rows, which tries to send back every single row. The summary is safe to collect on large data. The raw rows are not.

Common beginner questions

What architectural questions do beginners ask most often?

Here are three fundamental questions beginner data engineers encounter when learning distributed Spark architecture.

How is Spark different from pandas?

Pandas runs on one machine and holds all data in memory. Spark distributes data across a cluster of machines. Pandas is eager, while Spark builds lazy plans that execute on actions. Use pandas for small data and Spark for large data.

What is a partition?

A partition is a chunk of the dataset that one task processes. A DataFrame might be split into 200 partitions, each handled by a separate task on the cluster. More partitions allow more parallelism, but too many create scheduling overhead.

Why is lazy evaluation useful?

It lets Spark optimize the entire plan before executing. For example, Spark can push a filter close to the data source or skip unused columns before moving bytes across nodes.

Lazy evaluation allows Spark to optimize the full execution plan before touching a single byte on disk.

What comes next

What comes after understanding cluster architecture?

Now that you understand the architecture, you are ready to manipulate DataFrames directly: loading data, inspecting schemas, selecting columns, and filtering rows.

Distributed cluster execution lifecycle
  1. Driver coordinates the application and compiles lazy DAG plans.

    1. Python Script
    2. Logical Plan
    3. DAG Scheduler
  2. Cluster manager allocates executors across worker compute nodes.

    1. Cluster Manager
    2. Worker Nodes
    3. Executors
  3. Narrow operations pipeline; wide operations trigger network shuffles.

    1. Narrow Pipeline
    2. Shuffle Barrier
    3. Stage Boundary
  4. Executors aggregate results, returning bounded data to driver.

    1. Executor Tasks
    2. Partial Aggregation
    3. Safe Preview

In the next lesson, you will master the foundational vocabulary of Spark DataFrames and learn how to construct production data transformations.

  1. The driver coordinates and schedules; executors process partition tasks in parallel.
  2. Transformations build lazy plans while actions trigger physical execution.
  3. DataFrames give the Catalyst optimizer full visibility into schemas and operations.
  4. Never pull large tables to the driver with collect(); use bounded actions and aggregations.

Practice

Load orders with spark.table('orders'). Filter to paid orders. Instead of collect(), groupBy('order_status') and agg(F.sum('order_total').alias('total_paid_amount')) for a safe 1-row summary, then separately assign a 5-row limit of the filtered DataFrame to result.

The goal is practicing the distinction between a lazy plan (filter, groupBy) and the actions that trigger execution (agg's underlying count, show), while keeping bulk rows off the driver. The summary only runs when an action such as show() or collect() asks for it.

Going further: partitions are the unit of parallel work

The lesson says executors process partitions. A partition is a slice of the data, and Spark runs one task per partition, so the number of partitions decides how much work can happen at once. You can read that number and change it.

This program starts Spark on 2 local cores, creates 1,000 rows and then asks for 8 partitions. Predict the three numbers before you open the output.

PythonReading and changing the partition count
from pyspark.sql import SparkSession

spark = SparkSession.builder.master("local[2]").getOrCreate()

df = spark.range(1000)
print("partitions before:", df.rdd.getNumPartitions())
wide = df.repartition(8)
print("partitions after :", wide.rdd.getNumPartitions())
print("rows still       :", wide.count())
partitions before: 2
partitions after : 8
rows still       : 1000

The rows did not change. Only the way they are sliced did. With 2 cores, the first run had 2 slices and 2 tasks. After repartition(8) there are 8 slices, which means 8 tasks queue up for the 2 cores.

The partition count is a dial, not a fixed fact.

SituationWhat happensTypical fix
Very few partitions on a big clusterMost executors sit idleIncrease the partition count
Thousands of tiny partitionsTask start-up time dominates the workReduce the partition count
One huge partitionOne task is slow while the rest waitRepartition on a more even column

The number on your machine can differ, because the default depends on the cores available. The pattern is the same everywhere: read the count, then decide whether the work is spread evenly.

Try a new scenario

A job runs on 40 cores, but the Spark screen shows that only 4 tasks are running at a time, each taking 10 minutes. What do you check first?

Work it out, then open the answer

Check the number of partitions of the data being processed. With 4 partitions there are only 4 tasks, so 36 cores have nothing to do. Increasing the count, with repartition or by reading the data in more files, lets all 40 cores work at the same time. Do it before the expensive step, not after.

Does repartition cost anything?

Yes. It moves rows between executors across the network, which is a shuffle. It is worth it when it fixes badly uneven or too few partitions, and wasteful when the layout was already fine.

Close the lesson and explain it

Without scrolling up, explain how the number of partitions relates to the number of tasks, and why 4 partitions on 40 cores wastes the cluster.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?