Beginner friendly3 weeks

Data quality monitor for incoming files

Check every new file against rules and send a clear pass or fail report.

Tools: Python, pandas, Pydantic (optional), SQL, Email or Slack alerts (optional)

Why interviewers like it

Every company has bad data, and few students show they can catch it. A small tool that checks files and reports problems clearly is easy to finish, easy to demo, and exactly what hiring managers say they need.

The problem

Files arrive from other teams every day and sometimes they are wrong: duplicate ids, missing amounts, new category values. You will build a tool that runs a set of rules on each file and writes a report that says what failed and why.

How the data moves

  1. New file
  2. Load
  3. Run rules
  4. Report
  5. Alert or accept

The data

Use any public CSV, such as a sales or customer file, then corrupt a copy on purpose: duplicate some ids, blank some amounts, add an unknown status. Your tool must catch each problem you planted.

Week by week

  1. Week 1Pick a dataset, write 8 rules in plain English, and plant a bad copy of the file.
  2. Week 2Turn the rules into code: unique, not null, allowed values, ranges, row count, freshness.
  3. Week 3Produce a readable report, add a pass/fail exit code, and write the README with screenshots.

Starter code

Three rules, one clear report. Run it first, then grow it into the real project.

PythonThree rules, one clear report
import pandas as pd

df = pd.DataFrame({
    "order_id": [1, 2, 2, 4],
    "status": ["paid", "paid", "refunded", "unknown"],
    "amount": [500.0, None, 300.0, 120.0],
})

checks = {
    "order_id is unique": df["order_id"].is_unique,
    "amount has no nulls": bool(df["amount"].notna().all()),
    "status is an allowed value": bool(df["status"].isin(["paid", "refunded", "pending"]).all()),
}
for name, ok in checks.items():
    print(("PASS" if ok else "FAIL"), name)
print("failed:", sum(not ok for ok in checks.values()), "of", len(checks))
FAIL order_id is unique
FAIL amount has no nulls
FAIL status is an allowed value
failed: 3 of 3

This sample file was built to fail all three rules. Next, make the report list the failing rows too.

What to hand in

  • A command that takes a file and prints a pass/fail report
  • At least 8 rules covering uniqueness, nulls, values, ranges and counts
  • A good file and a bad file in the repository as test cases
  • A README with a sample report

Viva and interview questions

Practise saying each answer out loud. Short and honest beats long and vague.

Which checks did you choose and why?

Uniqueness on the key, not-null on important columns, allowed values for categories, a range for amounts, and a row-count check. These catch the most common real problems.

What should happen when a check fails?

It depends on the rule. Some failures should stop the load, and others should only warn. I made each rule's severity a setting.

How do you avoid noisy alerts?

I alert on failures that need action and keep warnings in the report. Too many alerts teach people to ignore them.

How did you test the tool itself?

With a clean file that must pass and a corrupted file where I know exactly which rules must fail.

How would you run it on every new file?

As a step right after the file arrives and before it is loaded, so bad data never reaches the tables people use.

How to stand out

  • Show failing rows in the report, not only counts
  • Save results over time to chart quality by day
  • Support rules in a YAML file so users can add checks without code

Put it on your CV

OutputResume bullet
Built a data quality monitor in Python that validates incoming files with configurable rules (uniqueness, nulls, allowed values, ranges, row counts) and produces pass/fail reports, tested against planted errors.

README outline

  1. The problem
  2. How to run it
  3. The rules
  4. Sample report
  5. How it was tested
  6. Ideas for more rules

Learn the pieces