Overview
Before any analysis, check the size, the columns, the missing values and the spread. Five commands that catch most surprises.
On this page5 sections
The table we keep using
You have been handed a new table. Before any question, spend two minutes looking. Surprises found now cost nothing. Surprises found after the report is sent cost a lot.
We use the orders table. Each row is one order, with a status and a total in rupees.
3 of the 180 rows.
| order_id | order_status | order_total |
|---|---|---|
| ORD-0001 | paid | 612.07 |
| ORD-0002 | refunded | 446.70 |
| ORD-0003 | paid | 1104.32 |
What we need from it
A first look answers four plain questions.
Four questions, four one-line commands.
| Question | Command |
|---|---|
| How big is it? | df.shape |
| Is anything missing? | df.isna().sum() |
| What categories exist, and how common is each? | df["order_status"].value_counts() |
| How spread out are the numbers? | df["order_total"].describe() |
Look before you compute. The first look is not optional.
Trace it step by step
Step 1: size and missing values
print(df_orders.shape)
print(df_orders.isna().sum())(180, 7) order_id 0 user_id 0 order_status 0 order_total 0 currency 0 created_at 0 updated_at 0 dtype: int64
180 rows, 7 columns, and zero missing values in every column. Write that down: "no nulls in orders" is a finding, not an absence of one.
Step 2: what categories exist?
print(df_orders["order_status"].value_counts())order_status paid 93 cancelled 30 pending 29 refunded 28 Name: count, dtype: int64
Only 93 of 180 orders are paid. If someone asks for revenue and you sum every row, you count cancelled orders too. That one mistake changes the answer by almost half.
Step 3: how spread out are the totals?
print(df_orders["order_total"].describe().round(2))count 180.00 mean 463.42 std 315.04 min 17.55 25% 197.17 50% 399.42 75% 684.96 max 1389.16 Name: order_total, dtype: float64
The smallest order is Rs 17.55 and the largest is Rs 1,389.16. The middle one is Rs 399.42. All amounts are positive, which is what you would hope for. A negative total would be a thing to ask about.
Same table, next cut
Look at the extremes
Sort, then look at the top and the bottom. This is where mistakes hide.
biggest = df_orders.sort_values("order_total", ascending=False).head(5)
print(biggest[["order_id", "order_total"]])order_id order_total 104 ORD-0105 1389.16 100 ORD-0101 1271.19 177 ORD-0178 1262.06 120 ORD-0121 1251.34 2 ORD-0003 1104.32
The biggest order is ORD-0105. The left number (104) is its row label in the table. It stays with the row when you sort.
Keep three lines in your notes: the size, what is missing, what the categories are. They are the first lines of your report.
Why is the mean 463.42 but the middle value 399.42?
A few big orders pull the mean up. The middle value ignores how big the biggest ones are. A later lesson covers this properly.
Should I fix problems I find in the first look?
Not yet. First note them. Cleaning decisions (drop, fill, flag) come after you know the whole picture.
What comes next
The first look checks one table on its own. Next you compare two tables that should tell the same story, and see whether they do.
Going further: spotting a column that is not what it seems
describe works on numbers. Text columns need a different check: how many different values do they have, and does that fit what the column should be?
import pandas as pd
orders = pd.DataFrame({
"order_id": ["O1", "O2", "O3", "O4", "O5", "O6"],
"user_id": ["U1", "U2", "U1", "U3", "U2", "U1"],
"status": ["paid", "paid", "refunded", "paid", "cancelled", "paid"],
"currency": ["INR"] * 6,
})
for col in orders.columns:
print(col, orders[col].nunique(), "different values in", len(orders), "rows")order_id 6 different values in 6 rows user_id 3 different values in 6 rows status 3 different values in 6 rows currency 1 different values in 6 rows
order_id has one value per row, which is what an ID should do. user_id repeats, because customers order more than once. status has just 3 values, a small list of categories. currency has exactly 1 value, so you never need to convert or group by currency here. The same check on the full orders table in the editor shows 180 order_ids, 74 customers, 4 statuses and 1 currency.
| Pattern | What it usually means |
|---|---|
| Distinct values equal the row count | An identifier. Safe to use as a key |
| A handful of distinct values | A category. Good for value_counts and groupby |
| Exactly one distinct value | A constant. Safe to ignore, but worth knowing |
| Many values, but fewer than rows | A repeated id such as a customer |
Try a new scenario
A new table has a column called country with 31 distinct values in 200,000 rows. Before running a single analysis, what would you check?
Work it out, then open the answer
Run value_counts and read the full list. 31 countries is plausible, but if the list contains "IN", "India" and "india", the same country is written three ways and any count per country is wrong. Fix the spelling first, then analyse.
Is a column with one value useless?
Not useless: it tells you something is constant. It becomes a warning if you expected variety, for example a currency column that should have several currencies.
Close the lesson and explain it
Without scrolling up, name the four questions a first look answers and the command for each.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.