Data engineering projects for beginners: 6 projects to build, in order

"Do you have any projects?" is one of the first questions in a fresher interview. It is a fair question. A certificate says you studied something. A working project shows that you can take real, messy data and turn it into something useful. This guide gives you six projects in an order that builds on itself, so each one teaches what the next one needs.

What makes a project good

• It starts with real data. Use a public dataset, not 3 rows you typed. Messy data is where you learn.
• It runs end to end. Raw data goes in, a useful result comes out, and anyone can run it by following your README.
• It has checks. Count the rows in and out, find duplicates, test totals. This separates a data project from a script.
• You can explain it. Why did you choose this design? What went wrong? What would you change? That last question is nearly always asked.

For data, look at public sources: the New York City taxi trip records, the Olist Brazilian e-commerce dataset on Kaggle, and open government data portals such as the one run by India's government. Pick something that interests you, because you will spend hours with it.

Project 1: A sales report in SQL

Goal. Answer 10 business questions about an orders dataset using only SQL.

Steps. Load a CSV of orders into DuckDB, SQLite or PostgreSQL. Write queries for revenue per month, top customers, best products, cancellation rate, and customers who never ordered. Save the useful ones as views.

Skills it shows. SELECT, JOIN, GROUP BY, HAVING, CTEs and window functions. If those are new, start with SQL for beginners and the joins guide.

In the README. List the 10 questions, the query for each, and the answer you found.

Project 2: A data cleaning script in Python

Goal. Turn a messy file into a clean one, and report what you rejected.

Steps. Take a CSV with real problems: missing values, inconsistent spelling, wrong types, duplicates. Write a script that reads it, cleans each field with small functions, writes the good rows to a new file, and writes the bad rows with a reason to a second file. Add a few tests for your cleaning functions.

Skills it shows. Python functions, the csv module or pandas, error handling, and honest reporting. See Python for data engineering beginners and pandas for beginners.

In the README. Show a before and after sample, the counts (rows in, kept, rejected) and the rules you applied.

Project 3: An API to database pipeline

Goal. Collect data from the internet every day and store it, safely.

Steps. Choose a free public API that needs no payment, such as weather, currency rates or public holidays. Write a script that calls it, saves the raw response, cleans it, and loads it into a table. Make the script idempotent: running it twice for the same day must not create duplicate rows.

Skills it shows. Ingestion, storing raw data, loading into a database, and safe reruns, as explained in what is a data pipeline.

In the README. Draw the flow, explain how you handle a day that fails, and show that a second run changes nothing.

Project 4: A small data warehouse

Goal. Design tables for analysis, not just storage.

Steps. Take an e-commerce dataset and model it as a star schema: one fact table of orders, and dimension tables for customers, products and dates. Load it with SQL, in raw, cleaned and reporting layers. Add checks, such as the row counts matching between layers.

Skills it shows. Data modeling and layering, from star vs snowflake schema, normalization and medallion architecture.

In the README. Include a diagram of your tables, the reason for each design choice, and three example questions the warehouse answers.

Project 5: A scheduled, orchestrated pipeline

Goal. Make project 3 or 4 run by itself, every day, and tell you when it breaks.

Steps. Run Airflow, for example in Docker on your own computer. Turn each step of your pipeline into a task. Set a daily schedule, add retries, and make one task fail on purpose to see the alert and the rerun.

Skills it shows. Orchestration, dependencies, retries and monitoring. Start from what is Apache Airflow.

In the README. Show a screenshot of a successful run and a failed one, and explain what happens when a task fails.

Project 6: A streaming demo (when you are ready)

Goal. See how events flow continuously.

Steps. Write a small Python program that produces fake order events. Send them to Kafka, run in Docker. Write a consumer that keeps a running total per city and prints it as events arrive.

Skills it shows. Topics, partitions, consumer groups and the difference between batch and streaming. Read what is Apache Kafka and batch vs stream processing first. This one is optional for a first job, so treat it as a bonus.

In the README. Explain why you chose each partition key, and what happens if the consumer restarts.

How to present your projects

1. 1Put every project on GitHub. One repository each, with a clear name.
2. 2Write a README that tells a story: the problem, the data, a diagram, how to run it, what went wrong, and what you would improve.
3. 3Keep the code tidy: small functions, no passwords in the repository, and a short list of steps to run it.
4. 4Prepare a 2-minute explanation of your best project, out loud, with no notes.

Common mistakes

• Copying a tutorial exactly. An interviewer can tell. Change the dataset or add a feature of your own.
• Using tiny fake data. Real, messy data is the point.
• No checks. Always count rows in and out.
• Too many tools. One project that uses three tools you understand beats one that uses 10 you do not.

There are more ideas, with plans, on the capstone projects page. To learn the skills each project needs, follow the Learn tracks, and read the data engineer roadmap for freshers to see where projects fit.