Lesson 2 of 3 in this module

7/21 in track

DataFrame basics

pyspark10 min

Overview

A Spark DataFrame looks like a table, but its rows may be spread across many workers. Start with SparkSession, read data, inspect the schema, select columns, and filter rows.

On this page8 sections

What you will do

A new engineer loads the orders table in Spark and tries to print the first row with the habits they learned in pandas. It works, but the data lives on many machines, and nobody can say which rows are where.

A Spark DataFrame looks like a table, but its rows are spread across workers. In this lesson you will load one, look at its columns, keep the columns and rows you need, and learn why nothing runs until you ask for a result.

What is a Spark DataFrame?

A DataFrame is a table with named columns, and every column has a type such as text, whole number or decimal. A pandas DataFrame and a SQL table work the same way from the outside.

The difference is where the rows live. In pandas all rows sit in one computer's memory. In Spark the rows are cut into chunks called partitions, and the chunks sit on many worker machines.

  1. Many partitions on workers
  2. Column names and types (schema)
  3. One table you can query

You still write code as if it were one table. Spark handles the splitting and the travelling for you.

How do you work with a DataFrame?

You use a few short methods. select picks columns, filter picks rows, and show prints a few rows so you can look at them.

  1. spark.table('orders')
  2. .select('order_id', ...)
  3. .filter(cancelled)
  4. show(10)

Each method returns a new DataFrame. The old one is never changed. That is why you can write the steps one after another and read them from top to bottom.

Before you change anything, look at what you loaded. printSchema lists every column with its type. show prints a few rows. Together they tell you what one row means.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
DataFrameA table that Spark spreads across machines
SparkSessionThe front door to Spark
SchemaThe column names and types of a table
selectChoose columns
filterKeep only the rows that match
LazyNothing runs until an action asks for a result

Why this skill

Why look at the data before you change it?

Every Spark job starts by loading a table. If you skip the look, you build on guesses.

For example, you might assume a status is always lowercase, but some rows say Paid. Or you might assume an id is a number, when it is text like ORD-0001. A wrong guess does not crash the job. It quietly drops rows or gives wrong totals.

Three quick checks before you build on a table

What to checkWhat can go wrongHow to check
Column typesA total stored as text cannot be added upprintSchema()
Sample rowsOdd casing or spelling in a valueshow(10)
Empty valuesRows silently dropped by a filterfilter(col.isNull())

These two habits, printSchema and show, come before joins, windows and everything else in Spark.

Why does Spark wait before it runs anything?

In pandas, a line like df[df.total > 100] runs at once. The rows are in memory, so the answer comes back straight away.

In Spark, select and filter do not touch the data. Spark only writes the step down in a plan.

Left: four code lines (table, select, filter, limit) drawn as a plan that reads no data. Right: result.show() runs the whole plan on workers and returns at most ten rows
Writing the steps is free. The work starts when you ask for a result.

Transformations build a plan. Actions run it.

A transformation is a step that makes a new DataFrame, such as select, filter or limit. An action is a request for a real result. show, count, collect and write are actions.

When you call an action, Spark looks at the whole plan, finds a cheaper way to run it, and sends the work to the workers. Because it saw every step first, it can skip work you did not need.

How the code works

How do you load a table?

RememberIf you know pandas, here is the translation

pd.read_csv('file.csv') becomes spark.read.csv(path). pd.read_parquet('file.parquet') becomes spark.read.parquet(path). df.head(10) becomes df.show(10). df.dtypes becomes df.printSchema(). The ideas are the same. The words differ.

spark.table("orders") loads a table by its name. A catalog, which is a list of table names and where their files live, tells Spark where to look. Most company platforms work this way.

To read files straight from storage, you use spark.read.parquet or spark.read.csv with a path. The previous lesson covers those.

Two routes to a DataFrame
Catalog table orstorage filesSparkSessionDataFrame planRun when you ask fora result

table() and read both give you a plan. Neither one reads the data yet.

How does printSchema show the column types?

printSchema prints the columns as a small tree. Each line is a column name, its type and whether it may be empty.

  • orders Schema TreeA schema tree like the one printSchema() prints
    • ├── order_id: string (nullable = true)
    • ├── order_status: string (nullable = true)
    • ├── order_total: double (nullable = true)
    • └── created_at: timestamp (nullable = true)

Look at order_total. It is double, which is a decimal number, so sum and avg will work. If it said string, the totals would be text and the maths would fail or give nulls.

When a type is wrong, fix it early with cast, before you build more steps on top.

Worked examples

How do you inspect a loaded table in Python?

Run the example below in this tab. Read the input, follow the code, then check the output matches what you expect.

PythonLoad the orders table and inspect its schema and first 10 rows
orders = spark.table("orders")

orders.printSchema()
orders.show(10, truncate=False)

PySpark-to-Pandas translation for basic DataFrame operations.

NeedPySpark expressionPandas equivalent
Load tablespark.table('orders')pd.read_parquet('orders.parquet')
Preview rowsdf.show(10)df.head(10)
Check schemadf.printSchema()df.dtypes
Select columnsdf.select('a', 'b')df[['a', 'b']]
Filter rowsdf.filter(condition)df.loc[mask]
Row countdf.count()len(df)

Notice how printSchema renders the column definitions while show prints the formatted preview grid.

How do select and filter narrow your DataFrame?

select keeps only the columns you name. Real tables often have dozens of columns, and most of them are not needed for one question.

With a Parquet table, Spark then reads only those columns from storage. Less data to read means a faster job.

filter keeps only the rows where a condition is true. where does the same thing under another name.

Both return a new DataFrame. The table you started from stays as it was.

PythonSelect 3 columns and filter to cancelled orders
from pyspark.sql import functions as F

result = (
    spark.table("orders")
    .select("order_id", "order_status", "order_total")
    .filter(F.col("order_status") == "cancelled")
    .limit(10)
)
result.show()
RememberFilter early, project early

Place filters and column selections as early as possible in the chain. Spark can push these operations toward the data source, reducing I/O and shuffle volume. This is the equivalent of adding a WHERE clause close to the FROM in SQL.

A null is an empty value. A filter such as order_total > 100 is neither true nor false for a row where order_total is null, so that row is dropped.

If you need to keep or find the empty rows, say so with isNull or isNotNull.

Why are strict data types necessary in production?

Parquet files remember the type of every column, so Spark gets the schema for free.

CSV and JSON files do not. If you let Spark guess, it reads the file once just to look, and the guess can change from run to run. For text files, write the schema yourself.

Schema makes the data useful
order_id: stringorder_status: stringorder_total: doubleORD-0001paid612.07ORD-0004cancelled339.90ORD-0005paid261.73

Named columns with types let Spark check your code before it runs.

The four types you will see most often

TypeWhat it holdsExample
stringTextORD-0001, paid
longWhole numbers3, 1042
doubleDecimal numbers612.07
timestampA date and a time2026-08-02 16:33:28

A type is a promise about every value in the column. Because of that promise, Spark can store the data compactly and warn you when a step does not make sense, for example adding up text.

How do you chain DataFrame operations cleanly?

Real Spark code chains the steps inside brackets, one step per line. It reads like a recipe, from top to bottom.

Give a DataFrame its own variable only when that step has a meaning of its own, such as valid_orders below.

PythonBuild a readable chain: select, filter, inspect
from pyspark.sql import functions as F

valid_orders = (
    spark.table("orders")
    .select("order_id", "order_status", "order_total")
    .filter(F.col("order_id").isNotNull())
    .filter("order_total >= 0")
)

valid_orders.show(5)
  1. Load through the catalog or a file reader.
  2. Check the schema and a small sample.
  3. Select only the columns you need.
  4. Filter invalid or irrelevant rows.
  5. Trigger an action to inspect or materialize.
Watch outshow() is still an action

A 10-row display looks cheap, but upstream joins, sorts, or aggregations still execute. Use show for inspection, not as evidence that the full plan is fast.

Common beginner questions

What common traps catch beginners when using DataFrames?

The biggest trap for people who know pandas is collect. In pandas all rows are already in your process, so reading them all is fine.

In Spark, df.collect() pulls every row from every worker onto one machine, the driver. On a big table the driver runs out of memory and the job fails with an OutOfMemoryError.

To look at a few rows, use show, take or limit. To save a large result, use df.write.

A second surprise: show(10) prints only ten rows, but Spark still runs every step before it. A cheap-looking show can sit on top of an expensive join.

Contrasting single-machine memory habits with distributed execution

OperationSingle-Node (pandas)Distributed (Spark)
df.collect()N/A (already in local RAM)Dangerous: pulls all cluster rows to driver RAM
df.head(N)Instant slice of local arrayTriggers DAG execution plan for N rows
df.filter()Computes boolean mask in RAMAppends filter operator to lazy Catalyst plan

Two questions come up again and again. First: what if the table is bigger than memory? pandas fails. Spark can move some data to local disk while it works, so many large jobs still finish. One huge partition or a collect can still run out of memory.

Second: is filter different from where? No. where is another name for filter.

What is the difference between select and filter?

select chooses columns vertically, while filter chooses rows horizontally. Use both to prune unnecessary data as early as possible.

Is show() the same as pandas head()?

Conceptually yes: both display a small sample. But show() is an action that triggers execution of the entire plan. In pandas, head() is instant because the data is already in memory.

What is column pruning?

When you select specific columns, Spark reads only those columns from Parquet files, skipping everything else. This dramatically reduces I/O for wide tables.

What happens if a DataFrame exceeds executor RAM?

Unlike pandas, Spark can spill shuffle and aggregation data to local executor disk, so many jobs larger than memory still complete. It is not a guarantee, because a single huge partition can still run out of memory.

What comes next

What comes after mastering basic DataFrame operations?

You can now describe a Spark DataFrame in one sentence: a table split across workers, where every step is only a plan until an action runs it.

Foundational DataFrame inspection and projection workflow
  1. Load table from catalog or path into lazy DataFrame plan.

    1. Storage / Catalog
    2. SparkSession
    3. DataFrame Plan
  2. Inspect column names, data types, and nullability with printSchema.

    1. printSchema()
    2. Column Types
    3. Nullability Check
  3. Project columns early to trigger Parquet column pruning.

    1. select(cols)
    2. Column Pruning
    3. Reduced I/O
  4. Filter rows early and inspect bounded sample with show.

    1. filter(condition)
    2. limit(10)
    3. show() Action

Next you will learn Column expressions: how to make a new column, branch with when and otherwise, and use Spark's built-in functions so your own Python never slows the job down.

Project columns early, filter rows close to the source, and let lazy evaluation optimize your pipeline end to end.

Practice

Load orders with spark.table('orders'). Select order_id, order_status, and order_total. Filter to cancelled orders. Limit to 10 rows. Assign to result and call show().

You are rehearsing the standard first pass: load, inspect, project, filter, sample.

Going further: why filter conditions need & and brackets

A filter condition on a DataFrame is a Column object, not a True or False value. That means the Python words and, or and not cannot combine conditions, and the symbols & | ~ need brackets around each part. Getting this wrong is one of the most common first errors.

The program below tries the Python word and, and then the right form. Predict which one raises an error.

Pythonand versus &
from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.master("local[2]").getOrCreate()

orders = spark.createDataFrame(
    [("A1", "paid", 900), ("A2", "paid", 120), ("A3", "cancelled", 900), ("A4", "refunded", 50)],
    ["order_id", "status", "total"],
)

try:
    orders.filter((F.col("status") == "paid") and (F.col("total") > 500))
except ValueError as exc:
    print(type(exc).__name__, "-", str(exc).split(":")[0])

orders.filter((F.col("status") == "paid") & (F.col("total") > 500)).show()
PySparkValueError - [CANNOT_CONVERT_COLUMN_INTO_BOOL] Cannot convert column into bool
+--------+------+-----+
|order_id|status|total|
+--------+------+-----+
|      A1|  paid|  900|
+--------+------+-----+

Python's and needs a plain True or False from each side, and a Column has none yet, because nothing has run. The symbol & builds a new combined Column instead, and Spark works it out later for every row.

The brackets matter because & binds tighter than == and >.

You wantWriteNot
Both conditions(a) & (b)a and b
Either condition(a) | (b)a or b
Not this condition~(a)not a
Compare first, then combine(F.col("x") == 1) & (F.col("y") > 2)F.col("x") == 1 & F.col("y") > 2

Try a new scenario

You need orders that are either cancelled or refunded, and also have a total above 100. Write the filter, and say what goes wrong if you forget the brackets around the first two conditions.

Work it out, then open the answer

Write ((F.col("status") == "cancelled") | (F.col("status") == "refunded")) & (F.col("total") > 100). Without the inner brackets, Python applies | before ==, so it tries to combine the text "cancelled" with a Column using |, and the result is an error or a wrong condition. Each comparison needs its own brackets.

Is there a shorter way for several values?

Yes. F.col("status").isin("cancelled", "refunded") tests one column against a list of values, so you only need & for the total check. It is shorter and harder to get wrong.

Close the lesson and explain it

Without scrolling up, explain why a Spark filter uses & instead of and, and why every comparison inside it needs its own brackets.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Practice this

Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.

Rate:
Was this useful?