Overview
Data engineers build the systems that collect, clean and store data, so analysts and models start from numbers they can trust.
On this page8 sections
The idea
Place an order on a shopping app and several internal services react in the same second. Different engineering teams own each service, so each service writes to its own isolated database.
What happens in the first second after you buy something?
When you tap Buy Now, four separate services immediately record that single action. The checkout app creates the order row, the payment service bills the card, the warehouse prints a packing slip, and the notification service emails a receipt.
- App databasestores order row
- Payment gatewaycharges card or UPI
- Warehouse serviceprints pick list
- Email servicesends receipt
Four separate systems now hold different pieces of the exact same purchase. Each service keeps its own data, so a slow stock query in the warehouse can never slow down checkout.
Every company runs many apps, but leaders need one combined truth.
Who connects those systems the next morning?
At 9 AM, leadership asks for total yesterday sales across all payment methods. None of the four operational systems can answer that alone.
Operational apps handle live users. Pipelines feed decision makers.
A data engineer builds the automated bridge between those worlds. Analysts building dashboards and data scientists training models depend on that single clean bridge.
Without that bridge, nobody can answer whether sales grew yesterday.
Why can we not run reports directly on the live checkout database?
A live database is tuned for quick single-row transactions, known as OLTP. A report that scans 5 million rows competes with checkout queries for CPU, memory and disk reads, so checkout can slow down.
OLTP powers live shoppers. OLAP powers analytical reports.
| System type | Workload name | Primary job | Query style |
|---|---|---|---|
| OLTP | Transaction processing | Fast inserts and single updates | Reads or writes 1 row by key |
| OLAP | Analytical processing | Aggregates across millions of rows | Scans entire columns for totals |
Running heavy analytical queries on the live database can slow down customers who are trying to pay.
Dedicated analytical stores called data warehouses (BigQuery, Snowflake, Redshift) are built for those big reports.
Never make a paying shopper wait behind a leadership report.
Words you will see in this lesson
Six words. The rest of the lesson uses only these.
| Word | What it means |
|---|---|
| Data engineer | The person who builds the pipes that carry data to where people can use it |
| Pipeline | Code that moves and cleans data on a schedule |
| ETL | Extract, transform, load: pull data out, clean it, store it |
| Warehouse | A database built for big reports |
| OLTP | Fast databases for live apps, one row at a time |
| OLAP | Databases built to add up millions of rows |
Why this exists
Systems written by different teams never format values identically.
How does the same sale look across different systems?
Here is one real transaction as three internal services stored it. Notice how each service picks its own format for money and time.
One purchase, three conflicting representations.
| System | Order id | Amount | Time |
|---|---|---|---|
| Website database | 1042 | 1299.00 | 2026-08-01 09:14 |
| Payment provider | txn_88f (ref 1042) | 129900 | 2026-08-01T09:14:07Z |
| Warehouse export | 1042 | (not recorded) | 02/08/2026 |
The website records rupees, the payment gateway stores paise as integers to avoid floating-point math errors, and the dates come in three formats. The warehouse export writes 02/08/2026, which could mean 2 August or 8 February.
Adding those raw amounts directly inflates company revenue 100 times.
What happens if nobody cleans this data with code?
Without an automated pipeline, each analyst cleans the mess manually in spreadsheets. Two analysts get slightly different numbers, and weekly meetings turn into arguments over whose spreadsheet is right.
Automated pipelines apply one rulebook nightly.
Automated code applies cleaning rules once and produces reliable tables for everyone. When new data lands tomorrow, the same rules run automatically on schedule without human intervention.
Clean data once in code so nobody debates numbers in meetings.
Picture this
Data travels a structured route from where it starts to where people use it.
What path does data travel from source to dashboard?
Every data pipeline follows three sequential responsibilities. The pattern is known as ETL: Extract, Transform, and Load.
Raw data enters on the left. Trustworthy reports leave on the right.
Extract pulls data from sources, transform standardizes fields, and load writes clean rows into the warehouse.
Dashboards and machine learning models query the clean warehouse, shielding live transactional applications from heavy analytical traffic.
ETL moves data from messy sources to trusted stores.
How does a data engineer differ from adjacent roles?
Data engineers build the foundation that other data teams rely on. Four roles collaborate in a modern organization, each handling a distinct step.
All three downstream roles depend directly on the tables delivered by data engineers.
| Role | Main daily focus | Core tools |
|---|---|---|
| Data engineer | Builds resilient pipelines and clean tables | Python, SQL, Spark, Airflow |
| Data analyst | Answers business questions and builds charts | SQL, BI tools, spreadsheets |
| Data scientist | Trains predictive models and tests hypotheses | Python, statistics, ML libraries |
| Analytics engineer | Builds business models and tests in warehouse | SQL, dbt |
If the data engineer delivers broken numbers, every downstream dashboard and model fails. When pipelines run smoothly, analysts spend time finding insights instead of fixing files.
A small example
Here is the smallest honest version of a complete pipeline.
How does a minimal pipeline run in Python?
Here are three raw orders from checkout handed over as text.
To Python these lines are plain strings with commas between fields.
| Raw text, one order per line |
|---|
| ORD-1,laptop,999.99,paid |
| ORD-2,mouse,24.99,cancelled |
| ORD-3,keyboard,79.99,paid |
Here is the Python script that extracts, transforms, and calculates revenue. You do not need to follow every line yet, because loops, split and try/except each get their own lesson. Read it for the shape: raw text goes in, clean rows come out.
raw_orders = [
"ORD-1,laptop,999.99,paid",
"ORD-2,mouse,24.99,cancelled",
"ORD-3,keyboard,79.99,paid",
]
clean_orders = []
for line in raw_orders:
parts = line.split(",")
clean_orders.append({
"id": parts[0],
"product": parts[1],
"price": float(parts[2]),
"status": parts[3],
})
paid_total = sum(o["price"] for o in clean_orders if o["status"] == "paid")
print(f"Paid revenue: {paid_total:.2f}")Predict the printed revenue before checking the output below: does it add the cancelled mouse?
Paid revenue: 1079.98
The filter skipped the cancelled 24.99 order, producing clean paid revenue.
clean_orders after the loop. Each price is now a validated number.
| id | product | price | status |
|---|---|---|---|
| ORD-1 | laptop | 999.99 | paid |
| ORD-2 | mouse | 24.99 | cancelled |
| ORD-3 | keyboard | 79.99 | paid |
Every line of code maps directly to a pipeline phase: split and float transform text, and clean_orders holds the loaded result.
How does the whole pipeline run from start to finish?
Extract: read raw text records from checkout services.
- Orders app
- Staging list
Transform: convert the text fields into numbers and keep the paid orders.
- Staging list
- Validator
- Clean rows
Load: write the clean records into the warehouse, in a way that is safe to repeat.
- Clean rows
- Warehouse
Three ideas to carry into the rest of this track:
- Data engineers build the automated systems that collect, clean, and deliver data for analytics.
- Operational databases (OLTP) serve users; analytical stores (OLAP) power heavy queries without slowing checkout.
- A pipeline has 3 jobs: extract the raw data, transform it into something trustworthy, and load it where people can use it.
Common beginner questions
Is data engineering the same as software engineering?
They overlap because both write code. Software engineers mostly build the apps users touch, such as mobile apps and websites. Data engineers build what happens to the data in the background. A good data engineer writes clean, tested, and reliable code.
Do data engineers build machine learning models?
Usually not. That is the data scientist's job. The data engineer ensures the model has fresh, correct data to learn from. If the training data feeding a model is corrupt, no amount of clever modeling will save it.
I have never programmed. Is that a problem?
No. This track starts from the very first variable. If you can use a computer and type, you have enough to begin. Each lesson runs real code in the editor on the page, so you learn by trying things.
What does a typical day look like for a junior data engineer?
You spend part of your day writing Python scripts to ingest data from vendor APIs or databases, part of your day writing SQL transformations in dbt or BigQuery, and part of your day monitoring pipeline health. When an overnight job fails or an upstream team changes a field name without notice, you inspect logs, locate the failure point, fix the pipeline, and backfill the missing data.
What comes next
Next you will see why Python became the standard language for building data pipelines.
After that we write code: variables first, then lists, dictionaries, files, and APIs. The SQL track covers the analytical warehouse end, and PySpark covers big distributed data.
Some topics from real pipelines are left for later, once you can write the code behind them: handling bad rows without crashing, making a rerun safe, the difference between ETL and ELT, and changes in upstream data. Each has its own lesson further along.
Practice
Run the sample code to watch the pipeline parse orders and calculate total revenue.
Then complete the exercise: take raw lines, filter for paid orders, and extract their prices accurately.
Going further: how do you know the bridge can be trusted?
A pipeline that finishes without an error is not the same as a pipeline that is right. The first question a data engineer asks after a load is: does the warehouse agree with the source? The cheapest way to ask is a reconciliation check.
Here is a source with 5 orders and a warehouse copy that looks fine at first glance. Predict which of the checks will notice the problem.
source = [("A1", 100), ("A2", 250), ("A3", 80), ("A4", 40), ("A5", 130)]
warehouse = [("A1", 100), ("A2", 250), ("A2", 250), ("A3", 80), ("A5", 130)] # A4 missing, A2 twice
def ids(rows):
return [order_id for order_id, _ in rows]
print("row count :", len(source), len(warehouse))
print("revenue :", sum(a for _, a in source), sum(a for _, a in warehouse))
print("missing ids :", sorted(set(ids(source)) - set(ids(warehouse))))
print("duplicate :", sorted({i for i in ids(warehouse) if ids(warehouse).count(i) > 1}))row count : 5 5 revenue : 600 810 missing ids : ['A4'] duplicate : ['A2']
Row counts match (5 and 5), so a count-only check would pass. The loss of A4 and the double load of A2 cancel out in the count. The revenue and the key comparison both catch it.
Use more than one check. Each one has a blind spot.
| Check | Catches | Can miss |
|---|---|---|
| Row count | Large losses and floods | A missing row hidden by a duplicate |
| Sum of a measure | Lost or doubled money | Two errors that cancel exactly |
| Key comparison (missing and duplicate ids) | Exactly which rows are wrong | Rows that exist but hold wrong values |
| Freshness (newest timestamp) | A load that silently stopped | Old rows that are wrong |
Try a new scenario
The app database says 4,000 orders were placed yesterday. The dashboard says 3,980. List the checks you would run, in order, and what each result would tell you.
Work it out, then open the answer
Start with the cheapest: compare revenue as well as the count, because a pure count can hide cancelling errors. Then compare the key lists to find the 20 missing ids. Look at their timestamps: if they all fall in the last few minutes of the day, the pipeline cut off early or used the wrong day boundary. If they are scattered, look for rows that were rejected by a validation rule, and check the quarantine table. Last, check freshness to see whether the job ran at all after the late orders arrived.
Why not just trust the job's success message?
A job reports whether it crashed, not whether the result is right. A load can finish cleanly after dropping rows, reading the wrong date, or writing a row twice. Checks against the source are what turn 'it ran' into 'it is correct'.
Close the lesson and explain it
Without scrolling up, explain why a matching row count does not prove 2 tables agree, and give one example from the check above.
Finished this lesson?
Mark it as read when you could explain it to someone else.