Overview
Good data is complete, accurate, unique, fresh, and valid. Count nulls on order_id to see completeness on a real warehouse frame.
On this page7 sections
The idea
Data quality is whether a table is fit for the decision it feeds. A revenue chart, a refund, and a fraud flag all need different columns to be trustworthy. The job of this track is to name those needs and check them before anyone publishes gold.
Five questions cover most of the work. Completeness: are required fields present? Accuracy: do values match reality? Uniqueness: is there one row per claimed grain? Freshness: is the newest row recent enough? Validity: do values sit in the allowed type and set?
This warehouse preloads real frames, not toy fixtures: df_orders, df_events, df_order_items, df_payments, df_customers, df_telemetry, df_customer_cdc. You will count, compare, and print. Later lessons wrap those counts in a dict shaped like {"success": bool, "failed_rows": int}.
Why this exists
An e-commerce shop that double-loads checkout events can report twice the revenue. Finance plans hiring and inventory on that number. The dashboard did not lie. It summed whatever landed in gold. Duplicate order_ids are a uniqueness failure that looks like growth.
A bank that stores transfers with a blank transaction_id cannot match a refund to the original debit. Completeness failed. The customer sees money leave and never come back. Support tickets pile up while the warehouse still shows a green nightly job.
Bad data is expensive because it is silent. Charts still render. Jobs still exit 0. The cost shows up as wrong GMV, duplicate customer records, and decisions made on yesterday's stock. Quality work is catching that before the wall screen, not after.
Picture this
Each layer is a different failure. Completeness is not the same as uniqueness.
Start with completeness because missing keys break everything downstream. If order_id is null, you cannot join payments, refund a row, or dispute a charge. pandas counts those blanks with isna().sum(). Wrap the numpy count in int() so the printed number is a plain Python int.
Name the dimension before you write the check. The helper you pick depends on which question you are asking.
| Dimension | Question | What goes wrong when it fails |
|---|---|---|
| Completeness | Is the required field present? | Joins and refunds have nothing to match |
| Accuracy | Does the value match reality? | GMV is the wrong amount even if every cell is filled |
| Uniqueness | One row per claimed grain? | SUM doubles; customers appear twice |
| Freshness | Is the newest row recent enough? | Inventory and fraud flags lag the real world |
| Validity | Is the value in the allowed set and type? | A status typo becomes a fifth bar on the chart |
Accuracy is harder to test from the table alone. Completeness, uniqueness, freshness, and validity can be measured from the frame. Accuracy often needs a second source: payments that should reconcile to order_total, or a quantity times unit_price that should match the header.
- Name the decision the table feeds (invoice, restock, fraud flag).
- Name the grain (one row per order, per event, per customer version).
- Pick the dimensions that would make that decision wrong.
- Count failures. Print the count. Later, wrap it in a success dict.
This warehouse's df_orders.order_id is complete: zero nulls. df_telemetry.reading is not. Sample prints both so you can see a passing count next to a failing one. The exercise grades the orders count.
- Completeness uses isna() on a required column.
- Uniqueness uses duplicated() on the grain column.
- Validity uses isin() against a named allowed set.
- Freshness uses a data clock such as max(event_time), not the job's finish time.
Print the count. Assign a DataFrame to result. The editor displays result in the grid. A lone integer in result is not a frame, so project at least one column, even when the interesting output is the printed number.
count = int(df_orders["order_id"].isna().sum())
print(count)
result = df_orders.loc[:, ["order_id"]]Sensor gaps are a different completeness story. Filling them with zero pretends the device reported a reading of 0. Leave them null, count them, and alert. The second example shows a column that is allowed to fail so you can see a non-zero count.
order_nulls = int(df_orders["order_id"].isna().sum())
reading_nulls = int(df_telemetry["reading"].isna().sum())
print("order_id nulls", order_nulls)
print("reading nulls", reading_nulls)
result = df_orders.loc[:, ["order_id"]]Zero is not a missing reading
fillna(0) on a measure can hide a completeness failure. A missing order_total is not a free item. Count nulls first. Fill only when the business rule says a blank means zero.
SQL IS NULL is pandas isna()
In SQL you wrote WHERE order_id IS NULL to hunt missing keys. In pandas, df[col].isna().sum() is that same hunt. The Quality track wraps the count in a printable dict. The pandas schema-contracts lesson used assert, which throws instead of reporting.
Before you start
This track assumes you completed Core Python and SQL & Warehousing. You will count nulls and duplicates on real warehouse frames with pandas.
A small example
Start with a slice of df_orders so you can see the grain column before you count nulls. Then compare it to df_telemetry.reading, where sensor gaps are allowed to fail.
Input: three orders. Every order_id is filled.
| order_id | order_status | order_total |
|---|---|---|
| ORD-0001 | paid | 84.20 |
| ORD-0002 | paid | 31.00 |
| ORD-0003 | cancelled | 12.50 |
count = int(df_orders["order_id"].isna().sum())
print("order_id nulls", count)
result = df_orders.loc[:, ["order_id"]]Every order_id in this warehouse slice is present, so the count is 0. A failing column would print a positive number instead.
Output: completeness on the key passes; the sensor column does not.
| column | null count | passes? |
|---|---|---|
| order_id | 0 | yes |
| telemetry.reading | > 0 | no (sensor gaps) |
Copy-paste without reading the output
Run Sample first. If the numbers or row count look wrong, stop and re-read the previous section before changing code.
Common beginner questions
Why not just look at the dashboard?
Dashboards aggregate. A duplicate key becomes a bigger bar, not an error. Completeness and uniqueness have to be checked on the grain column before SUM, COUNT, and GROUP BY hide the defect.
Is a null always bad?
No. Optional fields (a coupon code, a gift message) can be null. Required fields (order_id, a payment amount you will invoice) cannot. The dimension is completeness of what the decision needs, not completeness of every column.
What is the difference between accuracy and validity?
Validity is "this status is one of paid, cancelled, refunded." Accuracy is "this paid order really charged 84.20." Validity is a set check. Accuracy often needs a second table to reconcile against.
What comes next
The next lesson covers testing strategies: unit checks on one column, integration checks on joins, and contract checks that freeze what a consumer depends on. You will bundle two completeness and uniqueness checks into a tiny suite.
Practice
Run Sample to see a zero on order_id next to a non-zero on telemetry.reading. Then complete Exercise: count nulls in df_orders["order_id"] with isna, print the count, and assign result = df_orders.loc[:, ["order_id"]].
stdout should include 0. The code should contain isna. Do not assign the count to result.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.