Lesson 1 of 3 in this module

1/20 in track

What is pandas?

pandas10 min

Overview

Pandas is a Python library for tables. A DataFrame is the table. A Series is 1 column.

On this page8 sections

The idea

What does a morning at ShopKart look like?

A file lands with 180 online orders. Each has an id, a customer, a status like paid or cancelled, and a total.

Your manager wants the same cleaned-up version every morning, without anyone clicking through it by hand.

  1. Raw orders file
  2. Your pandas code
  3. Clean table

Pandas is a Python library, a bundle of ready-made code you borrow. You describe what to do to a table, and the computer repeats it.

What are a DataFrame and a Series?

Two words come up in every pandas lesson. This is the whole vocabulary you need to start.

3 words, one table.

WordWhat it isIn the orders file
DataFrameA whole table: rows down, named columns acrossdf_orders, 180 rows by 7 columns
Series1 column of a DataFramedf_orders["order_status"]
IndexA label for each row, 0, 1, 2 by defaultThe grey numbers down the left side

A DataFrame is the table. A Series is 1 column. The index is the row labels.

Why does ShopKart need this every day?

Thousands of purchases land every minute, each with a customer id, an amount, a discount code and timestamps.

  1. Midnight: the sales day ends
  2. Scheduled job copies order files
  1. Python worker runs pandas
  2. Clean names, add tax, flag fraud

An analyst can eyeball 50 rows. A pipeline cannot. Pandas does the cleanup in a script that runs unattended.

RememberBefore you start

This track assumes you finished Core Python. If lists, loops and functions still feel new, go through that track first and come back.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
pandasA Python library for working with tables of data
DataFrameA table with rows and named columns
SeriesOne column of a DataFrame
LibraryReady-made code you can borrow
RowOne record, such as one order
ColumnOne field, such as the amount

Why this exists

What does a plain Python total look like?

Say the orders arrive as dictionaries and you want the total of everything paid. You write the loop yourself.

PythonPlain Python: it works, but you write every step
orders = [
    {"order_id": "ORD-0001", "order_status": "paid", "order_total": 612.07},
    {"order_id": "ORD-0002", "order_status": "refunded", "order_total": 446.70},
    {"order_id": "ORD-0003", "order_status": "paid", "order_total": 1104.32},
]

paid_total = 0
for row in orders:
    if row["order_status"] == "paid":
        paid_total += row["order_total"]
print("Plain Python paid total:", round(paid_total, 2))

Predict: what does paid_total equal after the three turns of the loop?

Plain Python paid total: 1716.39

Fine for 3 orders. Filtering, sorting, joining and fixing typed text would each need another loop to write and debug.

Why is a pandas column faster than a list of dictionaries?

A list of dictionaries is many separate Python objects. Pandas keeps a whole column in one block.

Where the numbers live in memory
List of dictionariespandas columnmemoryEach dict is its own object with itsown hash tableEach float is a boxed object, 24 bytesObjects sit scattered, so the CPUchases pointersOne block of numbers side by sideA float64 takes exactly 8 bytesCompiled code sweeps it in one pass

Each Python float takes 24 bytes. A pandas float64 column takes 8 bytes per number.

Because the numbers sit side by side, a sum or a filter runs in compiled code instead of a Python loop. Modern CPUs can even work on several numbers per instruction.

What about a spreadsheet?

A spreadsheet is great for looking. But the steps live in your clicks, so nobody can rerun them tomorrow.

A formula dragged over the wrong range changes a report, and no history shows who did it.

Three places a table can live
Spreadsheet tab (click through it)pandas DataFrame (this editor)SQL table in a warehouse

The same rows and columns can sit in a spreadsheet file, in a pandas DataFrame inside Python, or in a SQL table in a warehouse. This track is about the middle one.

Pandas sits in between. The table is a real object in Python, one short line works on every row, and the line stays in your notebook.

Code can be read, tested, versioned in Git and rerun on a server. Clicks cannot.

Where does pandas stop?

Everything you load must fit in the RAM of one machine, and text loaded as Python objects takes several times the file size.

Measured with a generated orders-style file. Sorting or grouping then allocates temporary copies on top.

A 1 million row orders fileSize
CSV on disk64 MB
DataFrame, text as Python objects (older pandas)246 MB
DataFrame, text as the new str type (pandas 3)82 MB

Daily batches and sample-sized exports fit easily. For hundreds of gigabytes, teams move to PySpark, which splits the same work across many machines.

Picture this

What are the parts of a DataFrame?

A small orders table with the column names, the index, one highlighted column labelled Series, one dashed row and one highlighted cell.
The anatomy of a DataFrame, drawn with four real rows from the orders table you get in this tab.

Read it left to right. The grey strip is the index. The blue headers are column names. The green column is a Series. The dashed outline is one order. The orange box is one cell.

Why does the index travel with the rows?

The index is a label for each row, not just a position. Filter a table and each kept row keeps its label.

Predict: which row labels will the first three refunded orders carry?

PythonKeep refunded orders only
refunded = df_orders[df_orders["order_status"] == "refunded"]
result = refunded[["order_id", "order_status"]].head(3)

What result holds. The first column is the index.

order_idorder_status
1ORD-0002refunded
18ORD-0019refunded
20ORD-0021refunded

Labels 1, 18 and 20 stayed. Filtering never renumbers. If you want a plain 0, 1, 2 count, call df.reset_index(drop=True).

What is a dtype?

Every Series holds one kind of value, called its dtype. It decides how fast and how safely the math runs.

The four dtypes you will meet first.

dtypeHoldsIn df_orders
int64Whole numbersA count column, if you add one
float64Decimalsorder_total
object or strTextorder_status (str in pandas 3, object in older pandas)
categoryA few repeating labels, stored compactlyorder_status, if you convert it

What is already loaded in this tab?

You never write import pandas or open a file here. The studio built the tables for you.

The one we use most is df_orders: the 180 orders from the picture, with 7 columns.

The first row of df_orders.

ColumnExample
order_idORD-0001
user_idU004
order_statuspaid
order_total612.07
currencyINR
created_at2026-08-02 16:33:28
updated_at2026-08-02 19:33:28

To show a table, assign it to a variable named result. In real scripts you load with pd.read_csv or pd.read_parquet. After loading, every operation here works the same.

A small example

How big is the table?

Look before you touch anything. These are the first 3 rows of df_orders.

df_orders.head(3), showing four of its 7 columns.

order_iduser_idorder_statusorder_total
ORD-0001U004paid612.07
ORD-0002U073refunded446.70
ORD-0003U078paid1104.32

Predict: what shape and column list will df_orders give, and what type is a single column?

PythonAsk the table how big it is, what its columns are called, and what 1 column is
print(df_orders.shape)
print(df_orders.columns.tolist())
print(type(df_orders["order_status"]).__name__)
(180, 7)
['order_id', 'user_id', 'order_status', 'order_total', 'currency', 'created_at', 'updated_at']
Series

shape is a pair: 180 rows and 7 columns. 1 column pulled out with square brackets is a Series.

Check shape right after every load. A truncated file shows up as a wrong row count before bad data moves on.

RememberWorth remembering

If you only keep one thing from this lesson, keep the vocabulary: DataFrame is the table, Series is a column, index is the row labels. The rest is looking things up. In this tab the tables are already loaded, so you show one by assigning it to result.

How do you total the paid orders without a loop?

Here is the paid total from the loop above, written for pandas. Compare the two.

Predict: what will the sum of every paid order across all 180 rows be?

PythonNo loop, no running total
paid = df_orders["order_status"] == "paid"
print(df_orders.loc[paid, "order_total"].sum())
43606.63

Line one asks every row "are you paid?" and keeps the answers. Line two keeps the paid rows and adds up order_total.

What does that yes or no question actually build?

The comparison makes a column of True and False, one per row. It is called a mask.

The mask for the first 3 rows. The full mask has 180 entries, 93 of them True.

order_idorder_status== "paid"
ORD-0001paidTrue
ORD-0002refundedFalse
ORD-0003paidTrue

Passing the mask to .loc keeps only the True rows. The table did the looping for you.

How do you pick columns?

A list of names after the comma picks the columns. The colon before the comma means every row.

PythonPick 2 columns, keep 3 rows
result = df_orders.loc[:, ["order_id", "order_status"]].head(3)

What result holds. The first column is the index.

order_idorder_status
0ORD-0001paid
1ORD-0002refunded
2ORD-0003paid

head(3) trims the answer to 3 rows so the grid stays small.

Do one bracket and two brackets give the same thing?

No. One name gives a Series. A list of names, with brackets inside brackets, gives a DataFrame.

PythonOne name versus a list of names
one = df_orders["order_status"]
two = df_orders[["order_status"]]
print(type(one).__name__, one.shape)
print(type(two).__name__, two.shape)
Series (180,)
DataFrame (180, 1)

Some tools downstream need the two-dimensional shape, so know which one you are holding.

Why does my edit do nothing when I chain two brackets?

The first bracket picks rows and hands back a copy. The second bracket edits that copy, then it is thrown away.

Predict: do the cancelled totals change after the chained edit, and after the one-step .loc edit?

PythonTwo ways of trying the same edit
out = df_orders.copy()
out[out["order_status"] == "cancelled"]["order_total"] = 0
print(out.loc[out["order_status"] == "cancelled", "order_total"].head(3).tolist())

out.loc[out["order_status"] == "cancelled", "order_total"] = 0
print(out.loc[out["order_status"] == "cancelled", "order_total"].head(3).tolist())
[339.9, 549.7, 394.71]
[0.0, 0.0, 0.0]

The chained version changed nothing. Pandas 3 warns ChainedAssignmentError and older versions warn SettingWithCopyWarning, but the studio hides that warning, so nothing seems to happen. The one-step .loc version wrote real zeros.

Edit a selection in one step: .loc[rows, column] = value.

Why did my integer column turn into floats?

A native integer array cannot hold a missing value. One None changes the whole column's type.

Predict: what dtype does a Series of 101, 102 and None get?

PythonMissing values coerce integer columns to float64
ids = pd.Series([101, 102, 103])
print("ids dtype:", ids.dtype)

ids_with_na = pd.Series([101, 102, None])
print("ids_with_na dtype:", ids_with_na.dtype)
ids dtype: int64
ids_with_na dtype: float64

No warning appeared. Ids longer than about 15 digits can lose precision as floats, so pin id columns, for example dtype={'order_id': 'string'}.

Why avoid .iterrows()?

It runs, but it rebuilds every row as a fresh Series on each turn, throwing away the compact column layout.

PythonSame answer, very different work
total = 0
for _, row in df_orders.iterrows():
    if row["order_status"] == "paid":
        total += row["order_total"]
print(round(total, 2))

print(df_orders.loc[df_orders["order_status"] == "paid", "order_total"].sum())
43606.63
43606.63

Both agree. On 180 rows you will not feel the gap. On millions of rows, the mask version finishes far sooner.

How does the whole lesson replay on the running example?

The paid-orders total, step by step
  1. Start with df_orders: 180 orders in a DataFrame. Each column is a Series.

    1. df_orders
    2. 180 x 7
  2. Check shape first, so a truncated file shows up right away.

    1. df_orders.shape
    2. (180, 7)
  3. Ask every row if it is paid. That makes a mask of True and False.

    1. order_status == "paid"
    2. mask, 180 entries
  4. Keep the True rows and sum order_total. No loop, no running total.

    1. .loc[mask, "order_total"].sum()
    2. 43606.63

Five things to know from this lesson:

  1. A DataFrame is the table, a Series is 1 column, and the index is the row labels.
  2. Ask a question of a whole column and pandas answers for every row. Do not write loops.
  3. Filtering keeps the original row labels. Use reset_index(drop=True) for a fresh count.
  4. Edit a selection in one .loc step, never in two brackets.
  5. A missing value turns an integer column into float64, and the whole table has to fit in RAM.
IdeaA spreadsheet tab inside Python

A DataFrame is a spreadsheet tab. A Series is 1 column you have highlighted. The index is the row numbers down the left side.

Common beginner questions

Why can't I just use Excel?

For a quick look at a small file, you can, and you should. Pandas earns its place when you need the same cleanup to run again tomorrow on a new file, or when the table has to be handed to other Python code. The steps live in your notebook instead of your mouse clicks.

Do I need to install pandas in this tab?

No. The editor already has it, and the tables are preloaded. On your own laptop you would install it once with pip install pandas and then start your script with import pandas as pd.

What is the difference between a Series and a DataFrame?

A Series is 1 column. A DataFrame is several columns that share the same row labels. If a method needs a table with several named columns, give it a DataFrame. If it only needs one list of values, a Series is fine.

What comes next

Next you will look closely at the two shapes pandas uses: the Series (1 column) and the DataFrame (the whole table). You will also learn the two ways of picking rows and columns, .loc and .iloc, and why filtering is exactly where labels and positions stop lining up.

Practice

Open the exercise. You will take df_orders, keep two of its columns and the first 8 rows, and store that in result. Use df_orders.loc[:, ['order_id', 'order_status']].head(8) to select the columns and row slice. Check that the grid shows exactly the columns and the row count the prompt asks for.

Going further: will this DataFrame fit in memory?

pandas keeps the whole table in memory, so the first practical question about any file is: will it fit? You can answer it with arithmetic before you load anything. A number column stores 8 bytes per value, so size is roughly rows times number-columns times 8.

Predict the memory for 1,000,000 values, then for a table of 50 million rows and 10 numeric columns.

PythonEstimate before you load
import numpy as np

one_million = np.zeros(1_000_000)
print("1,000,000 numbers :", one_million.nbytes / 1_000_000, "MB")

rows, number_columns = 50_000_000, 10
gigabytes = rows * number_columns * 8 / 1_000_000_000
print("50M rows x 10 cols:", gigabytes, "GB of numbers alone")
1,000,000 numbers : 8.0 MB
50M rows x 10 cols: 4.0 GB of numbers alone

Eight megabytes per million numbers, and 4 GB for the big table before any text columns, before the index, and before pandas makes a temporary copy during a sort or merge. On a laptop with 16 GB, a 4 GB table is risky, because operations often need 2 or 3 times the table's size.

SymptomLikely causeWhat to try
MemoryError, or the machine starts swappingThe table plus its temporary copies exceeds RAMRead fewer columns (usecols), read in chunks, or move the work to SQL or Spark
Fits, but every step is slowPython-level loops over rowsUse vectorized column operations instead
Text columns use most of the memoryStrings are stored per valueConvert repeated labels to the category type

Try a new scenario

A CSV is 6 GB on disk with 40 million rows. You need the total revenue per country, and the file has 25 columns. What is your plan?

Work it out, then open the answer

Read only the 2 columns you need (usecols for country and revenue), and read in chunks, adding each chunk's per-country totals to a running total. Memory then depends on the chunk size, not the file size. If you will do this regularly or on many files, a query engine such as DuckDB or a warehouse does the same job without loading the table into pandas at all.

Why use pandas instead of a loop over rows?

pandas applies an operation to a whole column in compiled code, which is far faster than a Python loop, and it gives a clearer way to say what you mean. Loops over rows are slow and easy to get wrong.

Close the lesson and explain it

Without scrolling up, explain how to estimate whether a table fits in memory, and name two ways to work with a file that does not.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?