Python for data engineering beginners: what to learn first, with runnable code

SQL asks a database questions. Python is how a data engineer does everything around it: reading files, cleaning messy values, calling APIs, moving data from one place to another, and running the steps on a schedule. The good news is that you only need a small slice of Python to start. This guide covers that slice on a tiny example, and every output below came from really running the code in Python.

What data engineers use Python for

• Reading and writing files: CSV, JSON and others.
• Cleaning data: fixing text, converting types, handling bad values.
• Moving data: between files, databases and APIs.
• Glue: small scripts that run steps in order and check the results.

None of that needs advanced tricks. It needs lists, dictionaries, loops, if, functions, and a few modules from the standard library.

1A CSV, read into Python

Here is a small file of orders. In real life it would sit on disk, and you would open it. To keep the example runnable anywhere, the text is held in a variable and read as if it were a file. One order has a comma inside its amount, and one amount is not a number.

import csv
import io

raw = """order_id,customer,city,amount
1,Asha,Pune,"1,200"
2,Ravi,Mumbai,800
3,Meena,Chennai,n/a
4,Kiran,Pune,2500
5,Asha,Pune,300
"""

rows = list(csv.DictReader(io.StringIO(raw)))
print(len(rows), "rows")
print(rows[0])

5 rows
{'order_id': '1', 'customer': 'Asha', 'city': 'Pune', 'amount': '1,200'}

csv.DictReader turns each line into a dictionary: a set of name and value pairs, like {'customer': 'Asha'}. A list of such dictionaries is the most common shape for data in Python. Look carefully at the amount in the first row: '1,200' is in quotes. Every value from a CSV arrives as text, even the numbers.

2Why that matters: text is not a number

Try to add 1 to an amount, the way you would with a number:

rows[1]["amount"] + 1

TypeError: can only concatenate str (not "int") to str

Python refuses, because '800' is text and 1 is a number. You must convert first, with int(). But conversion can fail too:

int(rows[0]["amount"])

ValueError: invalid literal for int() with base 10: '1,200'

The comma in 1,200 makes int() give up. And the value n/a will fail the same way. Real data is full of these, which is why cleaning is most of the work.

3A function to clean one value

A function packages a piece of logic so you can reuse it and test it. This one removes commas and spaces and converts to a number. If the value cannot be converted, it returns None (Python's "no value") instead of crashing. try and except catch the failure.

def clean_amount(text):
    try:
        return int(text.replace(",", "").strip())
    except ValueError:
        return None

for sample in ["1,200", " 800 ", "n/a", "2500"]:
    print(repr(sample), "->", clean_amount(sample))

'1,200' -> 1200
' 800 ' -> 800
'n/a' -> None
'2500' -> 2500

All 4 samples behave sensibly: the good ones become numbers, and n/a becomes None. Testing a function on a few samples like this, before using it on real data, is a habit worth having from day one.

4Loops and filtering

Apply the function to every row with a for loop, and keep the good rows in a new list. Bad rows go to a separate list so you can look at them later, instead of vanishing:

clean_rows = []
rejected = []

for row in rows:
    amount = clean_amount(row["amount"])
    if amount is None:
        rejected.append(row)
    else:
        clean_rows.append({"customer": row["customer"], "city": row["city"], "amount": amount})

print("kept:", len(clean_rows))
print("rejected:", len(rejected))
print(rejected[0]["customer"], "had amount", rejected[0]["amount"])

kept: 4
rejected: 1
Meena had amount n/a

4 rows were kept and one rejected, and you know which one. Never quietly throw data away: a count of rejected rows is one of the first checks a pipeline needs.

5Summarise with a dictionary

To add up the amount per city, use a dictionary as a tally. get(city, 0) returns the current total, or 0 if the city has not been seen yet:

totals = {}
for row in clean_rows:
    totals[row["city"]] = totals.get(row["city"], 0) + row["amount"]

print(totals)

{'Pune': 4000, 'Mumbai': 800}

Pune has 1200 + 2500 + 300, and Mumbai has 800. Meena's Chennai order was rejected, so Chennai has no total at all. This is the same idea as GROUP BY city with SUM in SQL, which you can compare with the GROUP BY guide.

You can sort the result, largest first, with sorted and a key:

for city, total in sorted(totals.items(), key=lambda item: item[1], reverse=True):
    print(city, total)

Pune 4000
Mumbai 800

6A list comprehension: a loop in one line

A list comprehension builds a list from another in one line. It is the short form of a loop that appends. Here are only the Pune amounts:

pune_amounts = [row["amount"] for row in clean_rows if row["city"] == "Pune"]
print(pune_amounts)
print(sum(pune_amounts))

[1200, 2500, 300]
4000

It reads like English: "the amount, for each row, if the city is Pune". Use it for short, simple transformations, and a normal loop when the logic needs several steps.

7Write the result back out as CSV

The last step of a pipeline is to put the result somewhere. csv.DictWriter writes dictionaries as lines. Writing into a text buffer here keeps the example runnable anywhere, and in real life you would open a file instead:

out = io.StringIO()
writer = csv.DictWriter(out, fieldnames=["city", "total"], lineterminator="\n")
writer.writeheader()
for city, total in totals.items():
    writer.writerow({"city": city, "total": total})

print(out.getvalue().strip())

city,total
Pune,4000
Mumbai,800

8Putting it together: a 10-line pipeline

The same steps, as one small program built from functions. Each function does one job, which makes the program easy to read, test and change.

def read_rows(text):
    return list(csv.DictReader(io.StringIO(text)))

def clean(rows):
    good, bad = [], []
    for row in rows:
        amount = clean_amount(row["amount"])
        (bad if amount is None else good).append({**row, "amount": amount})
    return good, bad

def summarise(rows):
    result = {}
    for row in rows:
        result[row["city"]] = result.get(row["city"], 0) + row["amount"]
    return result

good, bad = clean(read_rows(raw))
print("good rows:", len(good), "| bad rows:", len(bad))
print(summarise(good))

good rows: 4 | bad rows: 1
{'Pune': 4000, 'Mumbai': 800}

You have just written a real data pipeline: read, clean, check, summarise. The tools get bigger (pandas for tables, Airflow for scheduling), but the shape stays the same.

Mistakes to avoid

• Forgetting that CSV values are text. Convert before you calculate.
• Letting one bad row crash the job. Use try and except, and count what you reject.
• Writing one long script. Split it into small functions you can test.
• Silent failures. Print or log how many rows went in, how many came out and how many were rejected.

What to learn next

1. 1Tables in Python: pandas for beginners.
2. 2The questions interviewers ask: Python interview questions for freshers.
3. 3Practice: the free Word Frequency Counter problem uses the same dictionary-tally idea, and the full Core Python track goes much further.