PySpark architecture: the one explanation

If you understand this properly, PySpark will stop feeling like magic. The verbs look familiar. The system underneath them is not a faster pandas. It is a distributed engine that plans work, splits data, and runs that work on many machines.

The biggest mistake beginners make is starting with this chain and never asking who actually executes it, where it runs, how data moves, and why Spark exists at all:

That line is a description of a computation, not a loop that walks rows on your laptop. Until you can picture the driver, the executors, the partitions, and the moment an action turns a plan into a job, the API will keep feeling like a black box. So let's build the entire picture from zero, in the same order you will meet it on a real cluster.

Imagine you have a Python program that lives entirely on one computer. It loads a list, loops through it, and builds a new list in memory. Nothing is distributed. One process does the whole job.

This works perfectly. Your computer processes the data. The list is tiny, RAM is plenty, and if the process crashes you just run it again. That is how almost everyone starts, and it is the right tool until the data outgrows the machine.

But what happens when your data becomes 10 GB, then 100 GB, then 1 TB, then 1 PB? A single machine starts having problems. Not one dramatic problem. Three separate ones that show up together: memory, speed, and failure.

Your laptop might have 16 GB of RAM. Your data is 500 GB. You cannot comfortably load everything into one machine. Even if the file is compressed on disk, parsing it into Python objects can be several times larger than the file itself. The process starts swapping, the kernel freezes, or you get an out-of-memory error in the middle of the loop.

You could try to stream the file line by line. That helps for simple filters. It falls apart the moment you need a group-by, a join, or a sort, because those operations need related rows to meet. Spark exists in part because those operations on large data need many machines' memory, not a cleverer loop on one laptop.

Suppose processing 1 TB takes one machine 10 hours. What if you had 100 machines? Ideally you divide the work, each machine processes a portion, and you finish much faster. Wall-clock time is what a nightly pipeline cares about. A job that finishes at 6am is useful. A job that is still running at noon is not.

You do not get a perfect 100x speedup. There is overhead to split the work, move data, and combine answers. The idea is still the reason Spark is worth running: many CPUs on many machines, each chewing a slice of the same job.

When you are processing huge data across many machines, something will fail. Disks die. Nodes get preempted. A JVM hits a bad record and exits. Your system needs to recover automatically, or every overnight job becomes a manual incident.

A single-machine script has a simple failure story: restart it. A 100-machine job cannot afford to throw away 99 finished slices because one worker died. Spark keeps enough history of how each slice was produced that it can recompute the lost piece instead of starting from zero. That is not a nice extra. It is why people trust Spark with production lakes.

Instead of sending the whole dataset into one computer's CPU and memory, we distribute the work. The data is split. Each machine sees a portion. The answers come back together. That shift, from one box to many, is the whole subject of this lesson.

We do this instead:

This is called distributed computing. The idea is simple, and you should be able to say it in one breath:

Everything else in Spark architecture (drivers, executors, partitions, shuffles, stages) is machinery for doing that split, that parallel work, and that combine, without you writing the networking yourself.

Apache Spark is a distributed data processing engine. Its job is to answer: you tell me WHAT you want to do with the data. I will figure out HOW to distribute and execute that work across multiple machines.

You stay in the language of tables: filter these rows, group by this column, sum this measure. Spark turns that description into tasks, puts those tasks on machines that have capacity, and brings the result back. You are not writing a cluster program. You are writing a query that a cluster can run.

You don't manually say which machine processes India, which processes USA, which processes UK, and who combines the results. Spark handles that. The moment you catch yourself thinking in machine numbers, you have dropped below the API Spark wants you to use. Think in tables and keys. Let Spark place the work.

Spark itself is primarily built using Scala and Java. The engine, the schedulers, the memory managers, and the query optimizer live on the JVM. PySpark is the Python API for Apache Spark. It is how Python programmers talk to that engine without writing Scala.

So when you write a group-by in Python, PySpark translates your instructions so that Spark's underlying distributed engine can execute them. Your Python process is not looping over every row. It is sending a description of the computation into Spark, then waiting for an action to demand a result.

That distinction matters the first time someone asks whether they should 'learn Spark or learn PySpark.' You learn Spark's architecture. You type PySpark. Same engine, Python spelling.

A cluster simply means multiple computers working together. Each one is still an ordinary computer. The cluster is the agreement that they share jobs, storage, and a resource manager. Nothing magical sits between them except the network and the software that assigns work.

Each machine can have CPU, RAM, and storage. Together they process large datasets. A four-machine cluster with 32 GB RAM each is not '128 GB of one computer.' It is four separate heaps, four sets of disks, and a network in between. Spark's whole job is to use those four heaps as if they were one processing system, while respecting that they are not one memory space.

Here is the architecture you should permanently remember. If you can redraw this from memory, the rest of this lesson is names for the boxes you already know.

There are four major things you need to understand: the Driver, the Cluster Manager, Worker Nodes, and Executors. Let's understand each deeply. If you mix these four up, every later topic (shuffle, skew, broadcast, cache) will feel like unrelated trivia. They are all consequences of this picture.

The Driver is the component to understand first. Think of Spark as a company. The Driver is the CEO, the brain. It does not necessarily process all the data itself. It decides what should happen, who should do it, and what 'done' looks like.

Instead, it receives your code, creates the execution plan, divides work into tasks, sends tasks to executors, monitors execution, and collects results. When a task fails, the Driver is the one that notices and asks for a retry. When you call show(), the small sample comes back to the Driver so it can print it.

Example. You write:

The Driver sees your instructions. Conceptually it lines them up as a sequence of steps, then creates a plan for executing that sequence across the cluster. You did not write those steps as a workflow engine. Spark inferred them from the DataFrame chain.

When working with PySpark, you usually start by building a SparkSession. That object is the handle your Python program holds for the rest of the application. Without it, you have no catalog, no readers, no SQL, and no connection to a cluster.

What is spark? It is your SparkSession. Think of it as your main entry point into Spark functionality. Architecture-wise, your Python code talks to SparkSession, SparkSession belongs to the Spark application on the Driver, and the Driver talks to the cluster.

Through SparkSession, you can read data, run SQL, and create DataFrames:

For beginners: SparkSession is how your PySpark application connects to Spark. getOrCreate() means 'reuse a session if this process already has one.' That is why notebooks often call it at the top of every cell without spawning a new application each time.

Imagine you have 100 machines. The Driver says 'I need 10 machines and 40 CPU cores.' Who decides which machines should be assigned? That is the job of the Cluster Manager. The Driver knows what it wants. The Cluster Manager knows what the cluster can spare.

Spark supports cluster managers such as Standalone, YARN, and Kubernetes. You will meet those names in job configs. The programming model does not change. Only the thing that hands out CPUs and memory changes.

The Cluster Manager says: okay. You can use Machine 1 with 4 cores, Machine 2 with 4 cores, Machine 3 with 4 cores. Then Spark starts executors on those machines. If the cluster is busy, you might get fewer cores than you asked for, or wait in a queue. That wait is not a Spark bug. It is the resource manager doing its job for every application on the cluster, not only yours.

Important distinction:

A Worker Node is simply a machine available in the cluster to perform Spark work. It is hardware (or a VM, or a pod) that can run executor processes. When people say 'the cluster has twenty nodes,' they mean twenty of these.

These are worker nodes. Inside worker nodes, Spark launches Executors. A worker can host one executor or several, depending on how the application is configured. Beginners can treat 'worker' as the machine and 'executor' as the Spark process on that machine. That split will keep you out of trouble when you read Spark UI later.

This is where actual computation happens. The Driver can be a brilliant planner. If no executor is running tasks, no data is processed. Executors are JVM processes that live on worker nodes for the life of your application (or until they are lost and replaced).

Example: the Driver fans work out to three executors. Each one processes a different slice of the data. They do not share Python lists in memory. They share a plan, and they exchange data only when a shuffle or a result collection requires it.

Executors perform the actual computation. They have CPU cores, memory, and tasks. A simple analogy: the Driver is the project manager, and the Executors are the employees doing the actual work. If you cache a DataFrame, it is executor memory that holds the cached partitions, not the Driver's laptop RAM.

If you remember only one comparison from this lesson, make it this table. Almost every production outage beginners cause is a Driver/Executor mix-up: collecting too much to the Driver, or assuming 'my Python variable' holds the lake.

Imagine you have a 1 TB file. Spark cannot efficiently give this entire dataset to one executor. Even if one machine had the RAM, you would throw away every other CPU in the cluster. The dataset has to be split into pieces that can be processed independently.

Instead, 1 TB of data gets divided into Partition 1, Partition 2, Partition 3, Partition 4, and so on. Each partition can be processed independently. Then Executor 1 can take Partition 1, Executor 2 takes Partition 2, Executor 3 takes Partition 3. This is how Spark achieves parallelism.

Those 250 GB numbers are only a sketch. Real partitions are often much smaller (128 MB or 256 MB is a common default from filesystems and Spark). The idea is the same: many piles, many tasks, many CPUs busy at once.

This sentence is extremely important: partitions determine how much parallel processing Spark can perform. Not the number of machines you rented. Not the number of lines of Python you wrote. The number of partitions that still have work left.

Imagine 100 GB of data in 10 partitions. Spark can potentially process multiple partitions simultaneously, one per task, spread across executors. But if you only have 1 partition, then one executor processes everything, even if you have 100 machines available. The extra machines sit idle because there is no extra pile to hand them.

So remember that chain. Data becomes partitions. Partitions become tasks. Tasks run on executors. If a job is slow and the cluster looks idle, the first question is not 'do we need more nodes?' It is 'how many partitions does this stage actually have?'

A task is the smallest unit of work sent to an executor. Let's say a dataset has four partitions. Spark may create four tasks: Task 1 processes Partition 1, Task 2 processes Partition 2, and so on. Then those tasks are scheduled onto executors that have a free core.

So a useful mental model is: Partition is the unit of data. Task is the unit of work. Typically, one task processes one partition for a particular stage. If a stage has 200 partitions, you will see about 200 tasks in Spark UI for that stage. That is not a coincidence. It is the same idea counted two ways.

Suppose you write a small pipeline: read a CSV, filter high salaries, count by department, then show the result. Let's trace the complete journey. This is the same story you will replay, with different column names, for the rest of your Spark career.

At this point, something surprising happens. Spark usually does NOT immediately execute the code. The assignments succeed. The Python variables exist. No cluster CPU has started scanning employees.csv yet. Why? Because Spark uses lazy evaluation.

When you write df.filter(...), Spark doesn't necessarily execute immediately. Instead, Spark says: okay, I remember that you eventually want to filter. Then groupBy(...): okay, I also remember that. Then .count() as a transformation chained on a grouped DataFrame: okay, noted. Spark builds a plan.

Only when you call an action, such as show(), collect(), count(), or write(), does Spark actually execute the work. Until that call, you have a recipe. You do not have cooked food. That is why printing a DataFrame object in Python shows a schema and a plan, not your rows.

Beginners fight this the first week. They add a filter, then stare at the cluster hoping to see CPU. Nothing happens. They think Spark is broken. Spark is waiting for an action. Once you expect that wait, lazy evaluation stops being a quirk and starts being the reason Spark can optimize.

Imagine you write three transformations in a row: filter salary, filter age, select a few columns. If Spark executed every line immediately, each operation would scan or materialize on its own. That could be inefficient. You would read columns you are about to throw away, and you would run two filter passes where one would do.

Instead, Spark waits. Then it sees the complete logic: read data, filter salary, filter age, select columns. Now Spark can optimize it. For example: I don't actually need all columns. Let me read only name, salary, and age. Or: let me combine these filters. This is one of the major reasons Spark can be efficient. Eager Python cannot easily do that, because each line already ran before the next line was written.

This leads to two extremely important categories. If you can sort a method into one of these two buckets, you can predict whether a line will hit the cluster.

Transformations describe what you want to do. They generally build the plan. filter, select, groupBy, join, withColumn, drop are the usual examples. When you write df2 = df.filter("salary > 50000"), no immediate execution necessarily happens. df2 is a new DataFrame object that knows how it would be computed.

Actions say: okay Spark, now execute. show, collect, count, write, take are the usual examples. When you write df2.show(), now Spark starts execution. The plan is compiled, stages are scheduled, tasks run, and a result (or a written dataset) appears.

A practical habit: count the actions in a notebook. Each one can re-run lineage from the start unless you cache. That is why a notebook with twelve show() calls on the same giant frame can cost twelve scans.

This is the architecture flow you should visualize. Keep it in your head as a vertical slide. Every later section is a zoom-in on one of these boxes.

Now let's understand every layer. You do not need to memorize JVM class names. You do need to know that Python is not the thing scanning Parquet, and that 'the DAG' is not a synonym for 'the job.'

DAG means Directed Acyclic Graph. Don't get scared by the name. Break it down, because each word is doing a job.

Directed: operations move in a direction, A to B to C. Acyclic: no circular loop. You never go A to B to C and back to A. Graph: a representation of connected operations. Spark internally builds this computational graph so it can see the whole recipe at once.

This execution relationship can be represented as a DAG. When Spark UI shows you a job, the picture you are looking at is this graph, sliced into stages. If you can sketch the DAG for a query, you can predict where it will shuffle before you run it.

Because Spark needs to understand what operations depend on other operations. You cannot aggregate by department until the filter has produced the rows that belong in the aggregate. You cannot filter until the read has produced rows. The DAG is that dependency list, drawn.

Spark creates this dependency graph. Then it can decide what can run together, what can run in parallel, where data movement is required, and how to recover if something fails. Without a DAG, retrying a failed task would mean guessing. With a DAG, Spark knows exactly which parent output a lost partition was built from.

This concept shows up in almost every slow Spark job. Let's start with partitions. Imagine three input partitions holding a few numbers each. A transformation either can finish inside each pile, or it cannot.

A narrow transformation means data from one input partition is used by only one output partition. filter, select, and map are the usual examples. Suppose you write df.filter("age > 25"). Spark can do Input P1 to Output P1, Input P2 to Output P2, Input P3 to Output P3. No need for partitions to communicate. This is efficient.

Narrow transformations pipeline well. Spark can run a read, a filter, and a select in one task, on one partition, without stopping to wait for any other executor. That is why a long chain of filters is usually cheap compared with a single groupBy.

A wide transformation means data needs to move across partitions. groupBy, join, orderBy, and distinct are the usual examples. Suppose we have India and USA rows split across two partitions, and we want groupBy("country"). Spark needs all India records together. But currently India lives in Partition 1 and also in Partition 2. So data must move.

This movement of data is called shuffle. Once you can see a shuffle in your head, Spark UI's 'Exchange' operator stops being a mystery. It is this picture, implemented with files and network transfer.

Shuffle means redistributing data across partitions. Example: df.groupBy("department"). Spark might need every executor to send some of its rows to other executors so that each department lives in one place. That is not a Python groupby on a local dict. It is a distributed regrouping.

Shuffle is expensive because it can involve network transfer, disk I/O, serialization, and sorting. This is why operations like groupBy, join, and orderBy can be expensive. They are not 'slow Python.' They are 'move a lot of data between machines.' When you optimize Spark, you are usually trying to do less of this, or to shuffle less bytes when you must.

Next comes how Spark cuts the DAG into stages. Spark sees the DAG and divides it into stages. Why? Because some operations can happen continuously without moving data. Others require a shuffle. Spark groups the first kind together, then draws a line at the shuffle, then starts a new group.

Spark might see Stage 1 as read, filter, select, and maybe a partial aggregation, then a shuffle, then Stage 2 as the final groupBy and aggregation. The shuffle creates a boundary. So: wide transformations usually create stage boundaries.

When Spark UI shows 'Stage 0' and 'Stage 1,' you are looking at this split. Stage 0 is not 'less important.' It is 'everything we can do before we have to move data.'

Beginners confuse these three constantly. Let's fix that forever. Suppose you filter, groupBy, count, and show. One Python statement chain. Several Spark nouns.

An action triggers a Job. show() creates a job. If you later call count() on the same DataFrame, that is a second job. Jobs are not 'your whole application.' They are 'one demand for a result.'

Spark divides the job into stages. Often two, separated by a shuffle boundary: Stage 1, then Stage 2. A job with two shuffles can have three stages. Count the wide boundaries, then add one.

Each stage gets divided into tasks. Stage 1 might have Task 1 through Task 4, one per partition. Each task processes a partition. The hierarchy is the thing to memorize:

This hierarchy is absolutely essential. Interviewers ask it because it is how you read Spark UI without drowning. Application is the whole program. Jobs are per action. Stages are per shuffle-free chunk. Tasks are per partition.

Let's use this code: read sales Parquet, filter amount over 100, group by country, sum amount, then show. Let's see what actually happens, from application start to the printed table.

You submit a PySpark Application. Spark creates a Driver. That is the process that will hold SparkSession, the DAG, and the schedulers. In local mode, that Driver is on your laptop. In cluster mode, it may be a process inside the cluster. Either way, one Driver per application.

The Driver says it needs resources. The Cluster Manager agrees (or queues you). Then executors start. You now have a Driver plus Executor 1, Executor 2, and Executor 3. Until executors register, there is nobody to run tasks. A job submitted in that window waits.

Suppose data is divided into four partitions under sales/. Spark knows the data can be processed in parallel. For Parquet, Spark can also see footer metadata: min/max values, column names, row-group sizes. That information feeds later optimizations. A CSV would give Spark much less to work with.

You wrote filter, groupBy, and sum. Spark builds something conceptually like Read, then Filter amount > 100, then Group by country, then Sum amount. Still no execution until show(). The Python variables result and the chain that produced it are plan objects.

show() is the action. Spark says: now I need to actually execute this. A Job is created. From this moment, Spark UI will show a job you can click into. Before this moment, Spark UI had an application and maybe some executors, but no job for this query.

For DataFrames and Spark SQL, Spark has something called the Catalyst Optimizer. Think of it as Spark's query optimizer. You tell Spark WHAT you want. Spark tries to decide HOW to execute it efficiently. That is the same split you already know from SQL databases: a query parser, then an optimizer, then an engine.

Example: you select two columns and filter a third. Spark may optimize with column pruning, predicate pushdown, filter reordering, and join strategies. Conceptually: your code becomes a logical plan, Catalyst rewrites it, you get an optimized logical plan, then a physical plan.

This is easier than it sounds. The Logical Plan describes what needs to happen: read sales, filter amount > 100, group by country, calculate sum. No detailed decision yet about exactly how. It is the query in operator form.

The Physical Plan describes how Spark will actually execute it. For example: Executor 1 reads partitions 1 and 2, Executor 2 reads partitions 3 and 4, they perform local filtering, shuffle by country, redistribute records, and perform aggregation. So: Logical Plan is WHAT. Physical Plan is HOW.

When you call explain(), you are asking Spark to print this HOW. You will practice that in a later lesson. For now, know that the physical plan is where join algorithms and shuffle operators show up.

The Catalyst Optimizer is mainly responsible for optimizing DataFrame and Spark SQL operations. Think: the developer says 'I want this result.' Catalyst looks for a better way to execute it. Then you get an optimized plan. You still wrote the same PySpark. The engine chose a cheaper path.

Common optimizations include the four below. You do not implement them. You write DataFrame code that Catalyst can see. Python UDFs, by contrast, are opaque, which is why they often disable this good luck.

Instead of reading everything and filtering later, Spark may push filtering closer to the data source when supported: read only relevant data. On Parquet and many lake tables, that can mean skipping entire row groups whose max amount is already below 100. On CSV, there is much less to push into. The filter still runs. It just runs after a full parse.

Suppose your dataset has 100 columns, but you only use name and salary. Spark can avoid unnecessarily reading all columns when the source format supports it. That is one of the practical reasons people store lakes as Parquet or Delta, not as giant CSVs.

Simplifies expressions where possible. Example conceptually: salary > (1000 + 500) can become salary > 1500. Tiny on its own. Together with the other rewrites, it keeps the physical plan from doing busywork.

Spark can choose different strategies for joining datasets. For example: Broadcast Join, Sort Merge Join, Shuffle Hash Join. We'll revisit joins later. The beginner takeaway is that Spark, not your Python, picks the join algorithm, using sizes and hints Catalyst can see.

Another term you'll hear: Tungsten. Catalyst focuses heavily on optimizing the query plan. Tungsten focuses heavily on making execution efficient. Historically, key areas include memory management, CPU efficiency, binary processing, and code generation.

You don't need to memorize all internal implementation details initially. Just remember: Catalyst optimizes the plan. Tungsten helps optimize execution. When someone says 'whole-stage codegen,' they are talking in the Tungsten neighborhood: generate tight code for a stage instead of interpreting one row at a time.

Our example filters sales then groups by country. Spark might create a job with Stage 1 as read data, filter amount > 100, and a partial aggregation, then a shuffle, then Stage 2 as the final aggregation. Why? Because groupBy("country") requires data to be reorganized. All records belonging to the same country need to come together.

That 'partial aggregation' line is worth a pause. Before shuffling, each partition can sum amounts per country locally. Then the shuffle moves much smaller partial totals, not every raw sale. Spark does this when it can. It is one reason a groupBy is expensive but not as expensive as shipping every row.

Suppose Stage 1 has 8 partitions. Spark creates approximately 8 tasks. Each task is 'run Stage 1's pipeline on this partition.' Executors process them. Depending on available CPU cores, multiple tasks can run simultaneously. An executor with one core runs one task at a time. An executor with four cores can run four.

Suppose Executor 1 has 4 CPU cores. Then it may run approximately 4 tasks simultaneously. Visualize four cores, four tasks. After one finishes, that core takes the next pending task. This continues until all tasks are complete.

This is why spark.executor.cores and the number of partitions interact. 10,000 tiny partitions on 4 cores means a lot of scheduling overhead. 2 giant partitions on 40 cores means 38 cores nap. Parallelism is a matching problem between piles of data and cores that can lift them.

Now combine everything. The user writes PySpark. SparkSession sits on the Driver. The Driver builds the DAG, optimizes, and schedules. The Cluster Manager grants workers. Executors run tasks in memory. Results flow back. This is the same diagram as section 6, now with the words you have earned.