Lesson 2 of 3 in this module

2/43 in track

What is data engineering?

python10 min

Overview

Data engineers build the systems that collect, clean and store data, so analysts and models start from numbers they can trust.

On this page8 sections

The idea

Place an order on a shopping app and several internal services react in the same second. Different engineering teams own each service, so each service writes to its own isolated database.

What happens in the first second after you buy something?

When you tap Buy Now, four separate services immediately record that single action. The checkout app creates the order row, the payment service bills the card, the warehouse prints a packing slip, and the notification service emails a receipt.

  1. App databasestores order row
  2. Payment gatewaycharges card or UPI
  1. Warehouse serviceprints pick list
  2. Email servicesends receipt

Four separate systems now hold different pieces of the exact same purchase. Each service keeps its own data, so a slow stock query in the warehouse can never slow down checkout.

Every company runs many apps, but leaders need one combined truth.

Who connects those systems the next morning?

At 9 AM, leadership asks for total yesterday sales across all payment methods. None of the four operational systems can answer that alone.

Operational apps vs data pipelines
Live applicationsData pipelinesgoalProcess one cart in millisecondsKeep isolated databases per servicePrevent checkout slowdownsPull rows from every serviceClean and align mismatched fieldsLoad one central reporting store

Operational apps handle live users. Pipelines feed decision makers.

A data engineer builds the automated bridge between those worlds. Analysts building dashboards and data scientists training models depend on that single clean bridge.

Without that bridge, nobody can answer whether sales grew yesterday.

Why can we not run reports directly on the live checkout database?

A live database is tuned for quick single-row transactions, known as OLTP. A report that scans 5 million rows competes with checkout queries for CPU, memory and disk reads, so checkout can slow down.

OLTP powers live shoppers. OLAP powers analytical reports.

System typeWorkload namePrimary jobQuery style
OLTPTransaction processingFast inserts and single updatesReads or writes 1 row by key
OLAPAnalytical processingAggregates across millions of rowsScans entire columns for totals

Running heavy analytical queries on the live database can slow down customers who are trying to pay.

Dedicated analytical stores called data warehouses (BigQuery, Snowflake, Redshift) are built for those big reports.

Never make a paying shopper wait behind a leadership report.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
Data engineerThe person who builds the pipes that carry data to where people can use it
PipelineCode that moves and cleans data on a schedule
ETLExtract, transform, load: pull data out, clean it, store it
WarehouseA database built for big reports
OLTPFast databases for live apps, one row at a time
OLAPDatabases built to add up millions of rows

Why this exists

Systems written by different teams never format values identically.

How does the same sale look across different systems?

Here is one real transaction as three internal services stored it. Notice how each service picks its own format for money and time.

One purchase, three conflicting representations.

SystemOrder idAmountTime
Website database10421299.002026-08-01 09:14
Payment providertxn_88f (ref 1042)1299002026-08-01T09:14:07Z
Warehouse export1042(not recorded)02/08/2026

The website records rupees, the payment gateway stores paise as integers to avoid floating-point math errors, and the dates come in three formats. The warehouse export writes 02/08/2026, which could mean 2 August or 8 February.

Adding those raw amounts directly inflates company revenue 100 times.

What happens if nobody cleans this data with code?

Without an automated pipeline, each analyst cleans the mess manually in spreadsheets. Two analysts get slightly different numbers, and weekly meetings turn into arguments over whose spreadsheet is right.

Manual cleaning vs automated pipeline
Without a pipelineWith a pipelinevsEvery analyst cleans by handReports disagree in Monday meetingsFixes vanish when new data landsRules written once in Python or SQLEveryone reads the same clean tableRuns automatically on schedule

Automated pipelines apply one rulebook nightly.

Automated code applies cleaning rules once and produces reliable tables for everyone. When new data lands tomorrow, the same rules run automatically on schedule without human intervention.

Clean data once in code so nobody debates numbers in meetings.

Picture this

Data travels a structured route from where it starts to where people use it.

What path does data travel from source to dashboard?

Every data pipeline follows three sequential responsibilities. The pattern is known as ETL: Extract, Transform, and Load.

The core pipeline path
Sources: app,payments, inventoryExtract: pull rawrecordsTransform: validateand convertLoad: write cleanrowsWarehouse:analytical storeDashboards: analystsand models

Raw data enters on the left. Trustworthy reports leave on the right.

Extract pulls data from sources, transform standardizes fields, and load writes clean rows into the warehouse.

Dashboards and machine learning models query the clean warehouse, shielding live transactional applications from heavy analytical traffic.

ETL moves data from messy sources to trusted stores.

How does a data engineer differ from adjacent roles?

Data engineers build the foundation that other data teams rely on. Four roles collaborate in a modern organization, each handling a distinct step.

All three downstream roles depend directly on the tables delivered by data engineers.

RoleMain daily focusCore tools
Data engineerBuilds resilient pipelines and clean tablesPython, SQL, Spark, Airflow
Data analystAnswers business questions and builds chartsSQL, BI tools, spreadsheets
Data scientistTrains predictive models and tests hypothesesPython, statistics, ML libraries
Analytics engineerBuilds business models and tests in warehouseSQL, dbt

If the data engineer delivers broken numbers, every downstream dashboard and model fails. When pipelines run smoothly, analysts spend time finding insights instead of fixing files.

A small example

Here is the smallest honest version of a complete pipeline.

How does a minimal pipeline run in Python?

Here are three raw orders from checkout handed over as text.

To Python these lines are plain strings with commas between fields.

Raw text, one order per line
ORD-1,laptop,999.99,paid
ORD-2,mouse,24.99,cancelled
ORD-3,keyboard,79.99,paid

Here is the Python script that extracts, transforms, and calculates revenue. You do not need to follow every line yet, because loops, split and try/except each get their own lesson. Read it for the shape: raw text goes in, clean rows come out.

PythonA minimal Python ETL pipeline
raw_orders = [
    "ORD-1,laptop,999.99,paid",
    "ORD-2,mouse,24.99,cancelled",
    "ORD-3,keyboard,79.99,paid",
]

clean_orders = []
for line in raw_orders:
    parts = line.split(",")
    clean_orders.append({
        "id": parts[0],
        "product": parts[1],
        "price": float(parts[2]),
        "status": parts[3],
    })

paid_total = sum(o["price"] for o in clean_orders if o["status"] == "paid")
print(f"Paid revenue: {paid_total:.2f}")

Predict the printed revenue before checking the output below: does it add the cancelled mouse?

Paid revenue: 1079.98

The filter skipped the cancelled 24.99 order, producing clean paid revenue.

clean_orders after the loop. Each price is now a validated number.

idproductpricestatus
ORD-1laptop999.99paid
ORD-2mouse24.99cancelled
ORD-3keyboard79.99paid

Every line of code maps directly to a pipeline phase: split and float transform text, and clean_orders holds the loaded result.

How does the whole pipeline run from start to finish?

The complete pipeline replay
  1. Extract: read raw text records from checkout services.

    1. Orders app
    2. Staging list
  2. Transform: convert the text fields into numbers and keep the paid orders.

    1. Staging list
    2. Validator
    3. Clean rows
  3. Load: write the clean records into the warehouse, in a way that is safe to repeat.

    1. Clean rows
    2. Warehouse

Three ideas to carry into the rest of this track:

  1. Data engineers build the automated systems that collect, clean, and deliver data for analytics.
  2. Operational databases (OLTP) serve users; analytical stores (OLAP) power heavy queries without slowing checkout.
  3. A pipeline has 3 jobs: extract the raw data, transform it into something trustworthy, and load it where people can use it.

Common beginner questions

Is data engineering the same as software engineering?

They overlap because both write code. Software engineers mostly build the apps users touch, such as mobile apps and websites. Data engineers build what happens to the data in the background. A good data engineer writes clean, tested, and reliable code.

Do data engineers build machine learning models?

Usually not. That is the data scientist's job. The data engineer ensures the model has fresh, correct data to learn from. If the training data feeding a model is corrupt, no amount of clever modeling will save it.

I have never programmed. Is that a problem?

No. This track starts from the very first variable. If you can use a computer and type, you have enough to begin. Each lesson runs real code in the editor on the page, so you learn by trying things.

What does a typical day look like for a junior data engineer?

You spend part of your day writing Python scripts to ingest data from vendor APIs or databases, part of your day writing SQL transformations in dbt or BigQuery, and part of your day monitoring pipeline health. When an overnight job fails or an upstream team changes a field name without notice, you inspect logs, locate the failure point, fix the pipeline, and backfill the missing data.

What comes next

Next you will see why Python became the standard language for building data pipelines.

After that we write code: variables first, then lists, dictionaries, files, and APIs. The SQL track covers the analytical warehouse end, and PySpark covers big distributed data.

Some topics from real pipelines are left for later, once you can write the code behind them: handling bad rows without crashing, making a rerun safe, the difference between ETL and ELT, and changes in upstream data. Each has its own lesson further along.

Practice

Run the sample code to watch the pipeline parse orders and calculate total revenue.

Then complete the exercise: take raw lines, filter for paid orders, and extract their prices accurately.

Going further: how do you know the bridge can be trusted?

A pipeline that finishes without an error is not the same as a pipeline that is right. The first question a data engineer asks after a load is: does the warehouse agree with the source? The cheapest way to ask is a reconciliation check.

Here is a source with 5 orders and a warehouse copy that looks fine at first glance. Predict which of the checks will notice the problem.

PythonReconcile source and warehouse
source = [("A1", 100), ("A2", 250), ("A3", 80), ("A4", 40), ("A5", 130)]
warehouse = [("A1", 100), ("A2", 250), ("A2", 250), ("A3", 80), ("A5", 130)]   # A4 missing, A2 twice

def ids(rows):
    return [order_id for order_id, _ in rows]

print("row count   :", len(source), len(warehouse))
print("revenue     :", sum(a for _, a in source), sum(a for _, a in warehouse))
print("missing ids :", sorted(set(ids(source)) - set(ids(warehouse))))
print("duplicate   :", sorted({i for i in ids(warehouse) if ids(warehouse).count(i) > 1}))
row count   : 5 5
revenue     : 600 810
missing ids : ['A4']
duplicate   : ['A2']

Row counts match (5 and 5), so a count-only check would pass. The loss of A4 and the double load of A2 cancel out in the count. The revenue and the key comparison both catch it.

Use more than one check. Each one has a blind spot.

CheckCatchesCan miss
Row countLarge losses and floodsA missing row hidden by a duplicate
Sum of a measureLost or doubled moneyTwo errors that cancel exactly
Key comparison (missing and duplicate ids)Exactly which rows are wrongRows that exist but hold wrong values
Freshness (newest timestamp)A load that silently stoppedOld rows that are wrong

Try a new scenario

The app database says 4,000 orders were placed yesterday. The dashboard says 3,980. List the checks you would run, in order, and what each result would tell you.

Work it out, then open the answer

Start with the cheapest: compare revenue as well as the count, because a pure count can hide cancelling errors. Then compare the key lists to find the 20 missing ids. Look at their timestamps: if they all fall in the last few minutes of the day, the pipeline cut off early or used the wrong day boundary. If they are scattered, look for rows that were rejected by a validation rule, and check the quarantine table. Last, check freshness to see whether the job ran at all after the late orders arrived.

Why not just trust the job's success message?

A job reports whether it crashed, not whether the result is right. A load can finish cleanly after dropping rows, reading the wrong date, or writing a row twice. Checks against the source are what turn 'it ran' into 'it is correct'.

Close the lesson and explain it

Without scrolling up, explain why a matching row count does not prove 2 tables agree, and give one example from the check above.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?