Lesson 1 of 2 in this module

1/12 in track

What is data quality?

pandas12 min

Overview

Good data is complete, accurate, unique, fresh, and valid. Count nulls on order_id to see completeness on a real warehouse frame.

On this page8 sections

The idea

The pipeline ran. It finished without an error. The dashboard updated. But the revenue number is wrong, and nothing told anyone.

That is the problem data quality solves. A table can load perfectly and still be wrong for the decision it supports.

Words you will see in this lesson

The five questions. Each one is explained below.

WordWhat it means
Data qualityHow well a table fits the job it is used for
CompletenessIs every required value filled in?
ValidityIs every value in the allowed set, format or range?
AccuracyDoes the value match the real-world fact?
UniquenessIs each real thing in the table exactly once?
FreshnessHow recent is the data compared with when things happened?

Good enough for the job

Data quality does not mean perfect. It means fit for purpose.

A table can be fine for counting daily orders and useless for checking a refund. Think of an inspector on a factory line. They do not ask if a part is flawless. They check it against the tolerance for that product.

5 questions before you trust a number
CompletenessAccuracyUniquenessFreshnessValidity

Each layer is a different way for the same table to be wrong. A table can pass four of them and fail the fifth.

Treat them as five separate questions. Passing one tells you nothing about the other four.

RememberBefore you start

This track uses filtering and isna() from pandas and NULL handling from SQL. We point to them in one line and do not teach them again.

Why this exists

A bug that doubled the revenue

An online shop has a bug. Every time a customer pays, the checkout event fires twice. Each order lands in the orders table two times.

The database accepts the rows. The job finishes. No error appears. Here is the table before and after, with three orders in rupees.

Correct: SUM(order_total) = 1280.00

order_idorder_total
ORD-0001850.00
ORD-0002310.00
ORD-0003120.00

After the double-fire bug: SUM(order_total) = 2560.00

order_idorder_total
ORD-0001850.00
ORD-0001850.00
ORD-0002310.00
ORD-0002310.00
ORD-0003120.00
ORD-0003120.00

The same query now shows twice the revenue. Every order_id appears twice. That breaks uniqueness: each order should be in the table exactly once.

An empty field can be just as costly

A bank records a transfer with an empty transaction_id. Weeks later a customer disputes it. Support cannot match the dispute to the transfer, because the key is missing.

That is a completeness failure: a field that must have a value is empty.

Bad data is dangerous because it fails quietly. Jobs finish. Tables update. Queries return rows.

Catching a bad value where it enters the pipeline takes one simple check. Finding it weeks later in a published report means re-running jobs and explaining the gap to everyone who used it.

Picture this

Five questions and what breaks

Name the question first. The check you write follows from it.

DimensionThe questionWhat breaks when it fails
CompletenessIs the required field filled in?Joins and refunds have no key to match on
ValidityIs the value in the allowed set, format or range?A status typo adds a strange category to a dashboard
AccuracyIs the value true in the real world?Revenue is wrong even though every cell holds a valid number
UniquenessIs each thing in the table once?Totals are doubled and customers get two emails
FreshnessHow old is the latest data?Stock counts and fraud alerts lag behind reality

Valid is not the same as accurate

Validity asks if a value follows the rules, like being a positive number. Accuracy asks if it is true.

An order total of 20.00 is valid. But if the customer was really charged 50.00, it is not accurate. To check accuracy you must compare with a trusted source, like the payment provider's records.

A passing check only says that one rule held on the rows present at that moment. It does not prove the data is correct.

A failing check says a rule was broken. It does not say why. Finding the cause is still your job.

IdeaA full shelf is not a true shelf

A shop shelf can look full while half the cartons are empty. Completeness asks if each carton has milk. Accuracy asks if the date on the carton matches what is inside.

Before writing a check, ask three things. What decision does this table support? What does one row mean? Which kind of failure would hurt that decision most? Completeness is usually the first to check, because a missing key breaks joins at once.

A small example

Six rows, four problems

This small orders table has six rows. Each bad row fails a different question.

Four bad rows, four different questions. Freshness belongs to the whole table, so no row shows it.

roworder_idstatusorder_totalverdict
0ORD-0001paid84.50good
1(empty)paid31.00bad: missing order_id (completeness)
2ORD-0003piad12.50bad: typo in status (validity)
3ORD-0004paid-45.00bad: negative total (validity)
4ORD-0005paid20.00good
5ORD-0005paid20.00bad: duplicate of row 4 (uniqueness)

Counting empty cells in pandas

In pandas, a missing value is None in text columns, NaN in number columns and NaT in date columns. isna() returns True for all three.

An empty string ("") or spaces (" ") is not null. To find those, you must check the text yourself.

Predict: how many empty order_id cells are there?

PythonCount empty cells in 1 column
import pandas as pd

orders = pd.DataFrame({
    "order_id": ["ORD-0001", None, "ORD-0003", "ORD-0004", "ORD-0005", "ORD-0005"],
    "status": ["paid", "paid", "piad", "paid", "paid", "paid"],
    "order_total": [84.50, 31.00, 12.50, -45.00, 20.00, 20.00],
})

missing = int(orders["order_id"].isna().sum())
print("order_id nulls", missing)
order_id nulls 1

Row 1 holds None, so the count is 1. int() turns the result into a plain Python number, which prints cleanly.

Running isna().sum() on every column gives a quick completeness report.

Completeness per column. order_total passes even though row 3 holds a bad negative value.

columnempty cellscomplete?
order_id1no
status0yes
order_total0yes

Try it on the warehouse tables

The Python editor in this tab already holds these tables: df_orders, df_events, df_order_items, df_payments, df_customers, df_telemetry and df_customer_cdc.

The order_id column in df_orders is complete. The reading column in df_telemetry is not, because sensors sometimes lose their connection.

PythonA clean key next to a gappy sensor column
print("order_id nulls", int(df_orders["order_id"].isna().sum()))
print("reading nulls", int(df_telemetry["reading"].isna().sum()))
order_id nulls 0
reading nulls 25

The orders table has no empty order IDs. The telemetry table has 25 missing readings. A count like this tells you which columns need extra care before you build on them.

Stop the run, or just watch

Checks work in two ways. A blocking gate stops the pipeline before bad rows reach the reporting tables. If order_id has nulls, stopping keeps broken join keys out of production.

Monitoring does not stop anything. It records numbers over time, like daily row counts or the share of nulls. It warns you when something starts to drift.

Tools such as dbt tests, Great Expectations and Soda turn these checks into automatic tests that run after every transformation.

A simple rule helps you choose. Stop the run when a wrong number could cause real harm, like a payment or a legal report. Only warn when the problem is cosmetic, like a spelling difference in a label that nobody joins on.

If you are not sure, start with a warning. Watch it for a week. If it never fires for a good reason, promote it to a gate.

Watch outZero is not a missing reading

Filling gaps with fillna(0) hides a completeness problem. A sensor that reports 0 measured zero. A missing value means no measurement arrived. Count the gaps first. Fill them only when the business rule says how.

Common beginner questions

Is data quality the same as data cleaning?

No. Quality checking measures whether the data is fit for use. Cleaning changes the data to fix problems. Always check first, because the real fix is sometimes in the source system and not in your table.

What does a passing check actually prove?

Only that the rule held on the rows present when the check ran. It does not prove that every expected row arrived, that the numbers are true, or that the table suits every purpose.

Why count the failing rows instead of stopping at the first one?

The count shows the size of the problem. One missing key in a million rows is a rare edge case. Five thousand missing keys suggest a break upstream.

From a quiet failure to five questions
  1. Pipeline: A scheduled pipeline finishes with exit code 0 and turns green. The data can still be wrong.

    1. exit 0
    2. green status
    3. data still wrong
  2. Duplication: An upstream bug fires checkout twice. Every order is duplicated and revenue doubles.

    1. double fire
    2. uniqueness fails
    3. revenue doubles
  3. Dimensions: Ask five questions before trusting a number: complete, valid, accurate, unique and fresh.

    1. complete
    2. valid
    3. accurate
    4. unique
    1. fresh
  4. Check: A check only proves that its rule held on the data it saw. It does not prove the data is clean.

    1. rule holds
    2. does not prove clean
  5. pandas: In pandas, isna() finds None, NaN and NaT. An empty string is not null and needs its own check.

    1. None/NaN/NaT = null
    2. "" != null
  6. Decision: Teams decide whether each check stops the run or only warns, based on who depends on the data.

    1. block or warn
    2. decision-driven

What comes next

Next lesson, Testing strategies for data, sorts checks by scope: checks on one column, checks across joined tables, and checks on the promises you make to other teams.

Practice

In the exercise, count missing values in df_orders["order_id"] using isna().sum(), print the count, and assign df_orders.loc[:, ["order_id"]] to result.

Going further: measuring the dimensions of quality

Data quality is not one thing. The common dimensions (completeness, validity, uniqueness, timeliness) can each be turned into a number, so quality becomes something you can track instead of a feeling. Predict which dimension looks worst for this small table.

PythonFour dimensions as four numbers
import pandas as pd

df = pd.DataFrame({
    "order_id": ["A1", "A2", "A2", "A4", "A5"],
    "email": ["[email protected]", None, "[email protected]", "not-an-email", "[email protected]"],
    "amount": [100, 250, 250, -5, 80],
    "loaded_at": pd.to_datetime(["2026-09-02 01:00"] * 4 + ["2026-08-30 01:00"]),
})
now = pd.Timestamp("2026-09-02 03:00")

print("completeness (email present):", df["email"].notna().mean())
print("validity (amount >= 0)      :", (df["amount"] >= 0).mean())
print("uniqueness (order_id)       :", df["order_id"].nunique() / len(df))
print("timeliness (loaded < 24h)   :", ((now - df["loaded_at"]) < pd.Timedelta(hours=24)).mean())
completeness (email present): 0.8
validity (amount >= 0)      : 0.8
uniqueness (order_id)       : 0.8
timeliness (loaded < 24h)   : 0.8

Each is the share of rows that pass: 80% have an email, 80% have a valid amount, 80% of order ids are distinct, and 80% were loaded recently. They are all 0.8 here, but they are different rows failing for different reasons.

A single quality score would hide that. Track the dimensions separately and set a target for each.

DimensionThe questionA simple measure
CompletenessIs the value there?Share of non-missing values
ValidityIs it an allowed value?Share inside the allowed range or format
UniquenessDoes each thing appear once?Distinct keys divided by rows
TimelinessIs it recent enough?Share of rows newer than the limit, or the age of the newest row
ConsistencyDo 2 sources agree?Share of keys whose values match across systems

Try a new scenario

A table of 1,000 rows has 950 emails present, 940 of which are valid, and 20 of the 1,000 order ids are repeats. State completeness, validity of the present values, and which numbers need a definition.

Work it out, then open the answer

Completeness is 95%. Validity of the present values is 940 of 950, about 98.9%, and note that you must decide whether to measure validity against all rows or only present ones. Uniqueness depends on how you count repeats: if 20 rows are extra copies, 980 of 1,000 are unique (98%), which is what the numbers 0.95 0.9895 0.98 show. Write down the definition next to each number so two people get the same answer.

What is data quality?

How well the data fits the use it is put to. Data can be perfectly fine for one report and unusable for another, so quality is measured against a purpose, using checks such as completeness, validity, uniqueness and freshness.

Close the lesson and explain it

Without scrolling up, name four quality dimensions and a number you could compute for each.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?