What is Apache Spark? A beginner's guide to how it works

You have learned pandas. It works well, until the day a file is so large that it will not fit in your computer's memory, or takes hours to process. You could buy a bigger computer, but there is a limit. The other answer is to use many computers at once, and Apache Spark is the most widely used tool for that. It takes a big dataset, splits it into pieces, spreads the pieces over many machines, and combines the answers.

Spark is written in Scala and runs on the Java virtual machine, and you can use it from Python (called PySpark), SQL, Scala or Java. The code in this guide is illustrative and was not run here, because Spark needs its own setup, so check the details against the documentation for your version.

The big idea: partitions

Spark splits a large dataset into smaller chunks called partitions. Each partition can be processed independently, so many can be processed at the same time, on different machines or on different cores of one machine. If you ask for the total sales, each partition adds up its own piece, and the pieces are combined at the end.

That is the whole trick. Everything else in Spark is about doing this splitting and combining well.

Driver, executors and the cluster

A Spark program has three kinds of part.

• The driver is the process that runs your program. It reads your code, works out the plan, and divides the work into small tasks.
• Executors are worker processes on the machines of the cluster. Each one runs tasks on the partitions it holds and sends results back.
• The cluster manager hands out the machines. On a cloud you rarely deal with it directly: services such as Databricks, Amazon EMR, Google Dataproc and Azure Synapse or HDInsight provide Spark clusters.

You can also run Spark on a single laptop, called local mode, which is how most people learn it.

Writing Spark code

In PySpark you work with a DataFrame, which looks like a table with named columns, much like pandas. Here is a typical beginner task: read a sales file, keep the large orders, and total them per customer.

df = spark.read.csv("sales.csv", header=True, inferSchema=True)

big_orders = df.filter(df.amount > 1000)
totals = big_orders.groupBy("customer").sum("amount")

totals.show()

If you know pandas or SQL this should look familiar: filter is WHERE, and groupBy with sum is GROUP BY with SUM. Spark SQL even lets you write the same thing as an actual SQL query.

Lazy evaluation: Spark waits

Here is the surprising part. When you run the first 3 lines above, Spark does almost nothing. It does not read the file or filter anything. It only writes down the plan. The work starts at the last line, totals.show().

• Transformations describe a new DataFrame from an old one: filter, select, groupBy, join. They are lazy.
• Actions ask for a result: show(), count(), collect(), writing to a file. They trigger the work.

Why wait? Because Spark sees the whole plan before running anything, so it can improve it, for example by reading only the columns that are needed and filtering as early as possible. If you ever wonder why a line of Spark code ran instantly and a later line took minutes, this is the reason: the slow line was the action.

Shuffles: the expensive part

Some operations are easy to do partition by partition, such as filtering rows. Others need rows with the same key to be brought together from every partition, such as groupBy, join and sorting. Moving data between machines like that is called a shuffle, and it is the most expensive thing Spark does, because the data travels over the network. Much of the skill of writing fast Spark code is avoiding unnecessary shuffles. You can see this in practice in the PySpark scenario guide, which covers broadcast joins and skew.

Spark, pandas or SQL?

A useful rule: do not use Spark for small data. Starting a Spark job has overhead, and for a few thousand or million rows, pandas or SQL is usually faster and simpler. Reach for Spark when the data truly needs many machines, or when your company's data platform already runs on it.

What to learn first

1. 1Get comfortable with tables and SQL: SQL for beginners and pandas for beginners. The DataFrame ideas carry straight over.
2. 2Learn the architecture ideas above, then write simple DataFrame code. Try the free Filter and Select Columns problem in the browser.
3. 3Move to joins, partitions and performance in the PySpark for Distributed Processing track, and the PySpark practice problems.