Lesson 3 of 4 in this module

6/20 in track

apply and map

pandas12 min

Overview

.map remaps a Series. .apply walks row by row or column by column. Vectorized comparisons are faster than both.

On this page8 sections

The decision

How many ways can you make a new column?

You have the orders table and you want a new column that says whether each order was paid. Pandas gives you three ways.

Two look at the values one by one in Python. The third hands the whole column to pandas in a single instruction.

Two ways to make the same column
Row by rowVectorizedvsapply(..., axis=1)A Python function runs per rowReads like a loopSlow as the table growsorder_status == "paid"One instruction for the columnReads like one sentenceStays fast as it grows

Row-by-row code asks a Python function about each order in turn. A vectorized comparison asks pandas once, and pandas answers for the whole column.

Both boxes produce the same column of True and False. What differs is how many times Python gets involved: once per row on the left, once in total on the right.

What do map, apply and vectorized each do?

Vectorized just means done on the whole column at once.

Three levers for a new column.

ToolWhat it doesExample
mapOn 1 column, replaces each value using a dictionary or a functionstatus.map({"paid": 1})
applyRuns a function you wrote, down a column or, with axis=1, across each rowdf.apply(fn, axis=1)
vectorizedA comparison, arithmetic between columns, or a .str methodstatus == "paid"

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
applyRun a function on each row or value
mapReplace each value using a lookup or a function
VectorizedOne operation applied to a whole column at once
FunctionA named piece of code you can run again and again
ColumnOne field of a DataFrame
LambdaA tiny function written on one line

What is at stake

How much slower is a row-by-row column?

On 180 rows you will not notice. On millions of rows the row-by-row version is the one that makes you fetch a coffee while it runs.

Predict: roughly how many times slower is apply with axis=1 than a comparison, on 200,000 rows?

Measured with pandas 2.2.3, the version in this tab, on 200,000 generated orders. Repeat runs gave a gap between 50 and 85 times. Your numbers will differ.

Way to make the columnTime for 200,000 rowsCompared with a comparison
order_status == "paid"about 8 ms1x
.apply(fn, axis=1)about 600 msabout 50 to 85x

The apply version returned the same answers. It just asked Python 200,000 separate questions to get them.

Same answer, tens of times the work. Speed is the price of row-by-row code.

What else does apply cost you?

It is more code to read and more to get wrong. A lambda you have to squint at is worse than a comparison that says exactly what it does.

A small example: df["order_total"] * 0.18 works on the whole column at once. df["order_total"].apply(lambda x: x * 0.18) gives the same answer but calls a Python function once per row. On 1 million rows, the first takes a blink and the second can take many seconds.

The habit you build now is what keeps later notebooks quick.

Option A vs Option B

Which tool should you reach for first?

From best default to last resort.

The decision table.

ToolWorks onTypical useVerdict
==, >, +, * on columnsA whole columnFlags and arithmeticFirst choice
.str.lower(), .str.strip()A text columnFix textFirst choice
np.where(test, a, b)A whole columnIf/else for each rowFirst choice
.map({...})1 columnTranslate codes using a lookupGood for lookups
.apply(fn, axis=1)Each rowLogic that needs Python per rowLast resort

The top 3 rows cover most of what beginners reach for apply to do. You only need apply when your rule needs a Python function with no pandas equivalent, such as calling another library for each row.

How do you decide in the moment?

Work down the questions in order and stop at the first yes.

Flowchart for making a new column: a comparison, arithmetic or str method leads to a vectorized expression, an if/else leads to np.where, a lookup dictionary leads to map, and everything else falls through to apply with axis=1 as a last resort
Work down the questions in order and stop at the first yes. apply only gets used when every question above it said no.Source: LakeBench · Original diagram

A worked comparison

What are we working on?

The first 5 orders, so you can see every row.

Input: df_orders.head(5).

order_idorder_statusorder_total
ORD-0001paid612.07
ORD-0002refunded446.70
ORD-0003paid1104.32
ORD-0004cancelled339.90
ORD-0005paid261.73

How do you make the flag the fast way?

Compare the whole column to the word paid, then assign the answer to a new column.

PythonCompare the whole column to the word paid
out = df_orders.head(5).copy()
out["is_paid"] = out["order_status"] == "paid"
result = out.loc[:, ["order_id", "order_status", "is_paid"]]
Output: a new True/False column
order_idorder_statusis_paidORD-0001paidTrueORD-0002refundedFalseORD-0003paidTrueORD-0004cancelledFalseORD-0005paidTrue

The highlighted is_paid column is the one we made. Every other column is untouched and the row count is still 5.

The comparison returns True or False for every row, lined up with the table. We called .copy() first so we edit our own table and not a view of the original.

What does the row-by-row version look like?

It gives the same answer and does far more work for it. Do not write this one.

PythonThe anti-pattern: apply across each row
out = df_orders.head(5).copy()
out["is_paid"] = out.apply(lambda row: row["order_status"] == "paid", axis=1)
print(out["is_paid"].tolist())
[True, False, True, False, True]

The same five answers as the fast way. The lambda ran once for each row, which is the slow part at scale.

What do you use for a lookup table?

Say a report wants numeric codes for the status instead of words. That is a lookup, and map is exactly for lookups.

Pythonmap swaps each value using a dictionary
out = df_orders.head(5).copy()
out["status_code"] = out["order_status"].map({"paid": 1, "refunded": 2})
result = out.loc[:, ["order_id", "order_status", "status_code"]]

Output. cancelled was not in the dictionary, so it became NaN (a missing value).

order_idorder_statusstatus_code
0ORD-0001paid1.0
1ORD-0002refunded2.0
2ORD-0003paid1.0
3ORD-0004cancelledNaN
4ORD-0005paid1.0

A value that is not a key in the dictionary becomes NaN, not an error. Because the column now holds a missing value, it turns into decimals: 1.0 and 2.0 instead of 1 and 2.

How do you stop a typo in the lookup from hiding?

Count the NaN values after a map. Then either add the missing key or fill on purpose.

Predict: with cancelled mapped to 3, what do the codes look like?

PythonA complete lookup leaves no NaN behind
out = df_orders.head(5).copy()
codes = {"paid": 1, "refunded": 2, "cancelled": 3}
out["status_code"] = out["order_status"].map(codes)
print(out["status_code"].isna().sum())
print(out["status_code"].tolist())
0
[1, 2, 1, 3, 1]

Zero NaN, and the column stays whole numbers. Check isna().sum() after every map on a real file.

How do you write an if/else column without apply?

People reach for apply to say "if this then that, otherwise something else". NumPy's where does it for the whole column.

NumPy is the number library pandas is built on, and it is already imported as np in this tab.

Pythonnp.where(test, value if true, value if false)
out = df_orders.head(5).copy()
out["size"] = np.where(out["order_total"] > 500, "big", "small")
result = out.loc[:, ["order_id", "order_total", "size"]]

Output: orders above 500 are big.

order_idorder_totalsize
0ORD-0001612.07big
1ORD-0002446.70small
2ORD-00031104.32big
3ORD-0004339.90small
4ORD-0005261.73small
Watch outapply with axis=1 is the first thing tutorials show, and the last thing you should reach for

If you catch yourself typing lambda row, stop and ask whether a comparison, an arithmetic expression, .isin, a .str method or np.where already does the job. In this lesson's exercise the checker rejects code that contains the word apply, even in a comment.

How does the decision replay on the running example?

Choosing a tool for the new column
  1. Start with the question: is each order paid? That is a comparison, so go vectorized.

    1. Need a paid flag
    2. comparison
  2. Compare the whole column once. One instruction, five answers.

    1. status == "paid"
    2. True, False, True, False, True
  3. Need codes from a lookup? Use map, and count the NaN values after.

    1. .map({...})
    2. isna().sum()
  4. Need if/else? Use np.where. Keep apply for what nothing else can do.

    1. np.where
    2. apply as last resort

Five things to know from this lesson:

  1. Vectorized means done on the whole column at once, and it is the default.
  2. Reach for a comparison, arithmetic or .str first, then np.where, then map, and apply(axis=1) last.
  3. apply(axis=1) gave the same answer as a comparison but took tens of times longer on 200,000 rows.
  4. map turns keys it does not know into NaN, so count isna() afterwards.
  5. Call .copy() before you add the column so the original stays as it was.
IdeaRoller or small brush

Vectorized code is like painting a wall with a roller: one pass covers the whole thing. apply is painting it one brick at a time with a small brush. Fine for a tiny wall, painful for a big one.

Common beginner questions

When is apply actually fine?

When each row needs something pandas cannot express, such as calling an outside library or parsing an odd format that no .str method handles. Try a vectorized version first, and use apply only when you are stuck.

What does axis mean?

axis=0, the default, means down a column. axis=1 means across a row. For a flag like is_paid you should not need either, because you should not need apply.

Is map faster than apply?

For a dictionary lookup on 1 column, map is the clearer and usually the better tool. For an equality flag, neither one is right. Use a comparison.

What comes next

Next you will see where pandas fits next to SQL. Filtering, grouping and joining exist in both, and picking the right one is mostly a question of where your table lives.

Practice

In the exercise you copy df_orders, add an is_paid column with a vectorized comparison, keep 3 columns and 12 rows, and store the result. The paid flag in the worked comparison is the same idea. Remember that the checker looks for the name is_paid and rejects the word apply anywhere in your code.

Going further: check that the fast version agrees with the slow one

The lesson compared apply with vectorized choices. Before you replace a slow apply with a fast column operation, prove they agree on your data, including the awkward rows. This habit is how you refactor without silently changing a report.

The rule: orders over 200 are 'large', otherwise 'small', and a missing amount is 'unknown'. Predict whether the 2 versions agree.

Pythonapply versus np.select
import numpy as np
import pandas as pd

df = pd.DataFrame({"amount": [50.0, 250.0, None, 200.0, 201.0]})

def label(amount):
    if pd.isna(amount):
        return "unknown"
    return "large" if amount > 200 else "small"

slow = df["amount"].apply(label)
fast = pd.Series(
    np.select([df["amount"].isna(), df["amount"] > 200], ["unknown", "large"], default="small"),
    index=df.index,
)

print(slow.tolist())
print(fast.tolist())
print("same:", slow.tolist() == fast.tolist())
['small', 'large', 'unknown', 'small', 'large']
['small', 'large', 'unknown', 'small', 'large']
same: True

They agree, including the boundary (200 is small, 201 is large) and the missing value. np.select reads the conditions in order and uses the first that is true, so the missing check has to come first.

If the NaN test came second, NaN > 200 is False and the missing amount would fall through to 'small'.

TaskToolWhy
Two outcomesnp.whereOne condition, 2 values
Several ordered rulesnp.selectConditions are checked in order
Replace values using a lookup tableSeries.map with a dictMissing keys become NaN, which is visible
Genuinely custom logic per rowapplySlowest, so use it last and keep it correct

Try a new scenario

Map the country codes IN, US and UK to their names, and see what happens to a code that is not in your table, 'DE'. Is that behaviour good or bad?

Open the answer

The result is ['India', nan, 'United States']: the unknown code becomes a missing value. That is good, because it is visible and countable. Silently keeping 'DE' would mix codes and names in 1 column, and silently using 'Unknown' would hide a gap in your lookup table. Count the missing values after a map as a check.

Interview: when is apply acceptable?

When the logic is genuinely custom per row and cannot be written with column operations, np.where, np.select or map. Keep it correct first, and measure before assuming it is a bottleneck.

Close the lesson and explain it

Without scrolling up, explain why np.select needs its missing-value test first, and how you check that a faster rewrite agrees with the original.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?