Lesson 2 of 3 in this module

5/43 in track

DE data structures

python12 min

Overview

Pipeline data is not one long list. Learn when to use lists, tuples, dictionaries, sets, and comprehensions to hold and reshape records.

On this page8 sections

What you will do

One order is easy to hold in your head. A day's worth is a thousand of them.

What three questions does one batch of orders raise?

The same data answers different questions, and Python gives you a different container for each. We use these three orders for the whole lesson.

The three orders we will use for the whole lesson.

idstatustotal
ORD-1paid120.5
ORD-2cancelled40.0
ORD-3paid88.25

Each question is a different way of touching the same data.

The questionThe container
Go through every order, in orderlist
Find the one called ORD-3dict
Which statuses exist at all?set
Keep one fixed record that must not changetuple

You will build all four, then finish with a one-line way to filter a batch.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
ListAn ordered collection you read by position
DictionaryA collection where you look up a value by its key
KeyThe label you look things up by
TupleA list that cannot be changed after it is made
SetA collection with no repeats
LookupFinding a value by its key

Why this skill

Picking the right container decides how fast and how clean your code is.

What happens if you pick the wrong container?

The code still runs, but it gets slower or quietly wrong. Collect ids in a list when you meant to deduplicate and the duplicates stay.

Four containers, four jobs
Ordered, in a rowLooked up or checkedpick by joblist: ordered batch you can changetuple: fixed record, cannot changedict: find a value by its keyset: unique values only

Choose by how you will use the data, not by how it looks.

Left side: position or fixed shape matters. Right side: you ask is this one here, or what goes with this key.

Why is a dictionary lookup so much faster?

Searching a list with in scans from the first item to the last. A dictionary or set jumps straight to the answer.

The difference between a pipeline that finishes in seconds and one that hangs for hours.

listdict or set
How it finds ORD-3Checks items one at a timeComputes where the key lives
With 1 million rowsUp to 1 million comparisonsAbout the same work as with 10 rows
Cost grows with sizeYes (O(N))No (O(1) on average)

The right container can turn a slow step into a fast one, as the next example shows.

A list searches by walking. A dict searches by jumping.

What does that speed buy on a real join?

ShopKart has 100,000 evening orders and 50,000 customer records. Each order needs its customer contact details.

Same answer, very different compute cost.

ApproachWork done
Nested loops: scan all customers for every order100,000 x 50,000 = 5 billion comparisons
Dict of customers, then one lookup per order50,000 + 100,000 = 150,000 operations
  1. Loop through the 50,000 customers once and build a dictionary keyed by customer_id.
  2. Loop through the 100,000 orders once and do one dictionary lookup for each customer_id.

Nothing changed except the data structure, yet performance improved by thousands of times.

How the code works

Python provides clean syntax to construct, query, and transform each container. The examples use for and if to walk through the data. Those get their own lesson next, so for now read them as plain instructions: for each order, do this.

How does a list keep order?

Square brackets make a list. Items keep the order you gave them, and a for loop visits them one by one. Our orders are a list of dictionaries. Predict the print order.

PythonWalk the whole batch in order
orders = [
    {"id": "ORD-1", "status": "paid", "total": 120.5},
    {"id": "ORD-2", "status": "cancelled", "total": 40.0},
    {"id": "ORD-3", "status": "paid", "total": 88.25},
]

for row in orders:
    print(row["id"], row["total"])
ORD-1 120.5
ORD-2 40.0
ORD-3 88.25

Three orders, three lines, in the exact sequence they were stored.

How does a dictionary find a value by key?

Curly braces with key: value pairs make a dictionary. Each order above is one, and row["total"] reads the value stored under that key.

A dictionary can also index the whole batch. Here we key it by order id, so we can jump directly to ORD-3. What does a missing key give?

PythonLook up by id
by_id = {row["id"]: row for row in orders}

print(by_id["ORD-3"])
print(by_id.get("ORD-9"))
{'id': 'ORD-3', 'status': 'paid', 'total': 88.25}
None

ORD-9 is not in the batch, so .get() gave None. Writing by_id["ORD-9"] would crash with a KeyError.

A dictionary: key in, value out
ORD-1paid, 120.5ORD-2cancelled, 40.0ORD-3paid, 88.25

Ask by key and you get the record straight away, however many entries there are.

Left boxes are keys and right boxes are records. Python does not read down the keys to find yours.

Use .get() when a key might be missing. Use brackets only when its absence should stop the job.

What is actually inside a dictionary?

A hash function turns each key into a bucket number. The value sits in that bucket.

A hash table drawn as a phone book: the keys John Smith, Lisa Smith and Sandra Dee go through a hash function box and each lands in a numbered bucket holding a phone number
Each key goes through a hash function that picks a bucket, and the value sits in that bucket. This is what sits underneath a Python dict.Source: Wikimedia Commons, Jorge Stolfi · CC BY-SA 3.0

To look a name up again, Python runs the same function and walks straight to that bucket. Swap names for order ids and phone numbers for records and you have by_id.

How does a set keep each value once?

A set has no duplicates and no order. It answers which different values exist. Predict the counts of ids before and after.

PythonUnique values, and counting duplicates
statuses = set(row["status"] for row in orders)
print(sorted(statuses))

ids = ["ORD-1", "ORD-2", "ORD-1"]
print(len(ids), len(set(ids)))
['cancelled', 'paid']
3 2

The set collapsed three ids into two. That count difference is how you detect duplicates in a raw feed.

We wrapped the first set in sorted() because a set has no order of its own and may print differently each run.

How do sets find missing orders?

The website says it took 10,000 orders and the warehouse says it packed 9,985. Where are the 15? Set operations answer in one line each.

PythonDifference, intersection and union
placed = {"ORD-1", "ORD-2", "ORD-3"}
packed = {"ORD-1", "ORD-3"}

print(sorted(placed - packed))
print(sorted(placed & packed))
print(sorted(placed | packed))
['ORD-2']
['ORD-1', 'ORD-3']
['ORD-1', 'ORD-2', 'ORD-3']

Difference is placed but not packed. Intersection is in both. Union is everything seen at least once.

What makes a tuple a fixed record?

Round brackets make a tuple. It looks like a list, but once it exists you cannot change it. Can you edit grain[0]?

PythonA tuple as a record, and as a dictionary key
grain = ("ORD-1", "2026-08-01")
print(grain[0])

sales = {grain: 120.5}
print(sales[("ORD-1", "2026-08-01")])

grain[0] = "ORD-9"
ORD-1
120.5
TypeError: 'tuple' object does not support item assignment

Python refuses the edit. Because a tuple can never change, it can be a dictionary key, which a list cannot.

Why can a tuple be a key but a list cannot?

A list can change in place, which would move its bucket and lose the key. A tuple cannot, so it is hashable.

A composite key such as warehouse plus SKU tracks stock per facility.

KeyWorks?Why
("BLR_WAREHOUSE", "SKU_7729")YesImmutable, so its bucket never moves
["BLR_WAREHOUSE", "SKU_7729"]NoMutable lists are not hashable

Records with compound primary keys are common, and tuples are how you key a dictionary by several columns.

Which container do you reach for?

Flowchart: what will you do with the values? Walk through them in order leads to list, find one by a key leads to dict, see which different values exist leads to set, keep a fixed record leads to tuple
Start from what you will do with the data, then follow the arrow.Source: LakeBench · Original diagram

Start from the question and follow the arrow. If two arrows seem to fit, use a list, the one you will change your mind about least.

Worked examples

Example 1: how does a comprehension replace a loop?

Picking the paid totals takes a loop that builds a new list. A comprehension does the same in one line. Predict what is in paid_totals.

PythonLoop first, comprehension second
paid_totals = []
for row in orders:
    if row["status"] == "paid":
        paid_totals.append(row["total"])

# the same thing in one line
paid_totals = [row["total"] for row in orders if row["status"] == "paid"]

print(paid_totals)
print(sum(paid_totals))
[120.5, 88.25]
208.75

Read it from the middle out. for row in orders walks the batch, the if keeps the paid ones, and row["total"] at the front says what to collect.

Example 2: how do you read a nested payment event?

API responses are dictionaries inside dictionaries, with lists inside those. You walk in one key at a time.

PythonWalking nested data
event = {
    "user": {"id": "U001", "country": "IN"},
    "items": [{"sku": "A1", "qty": 2}],
}

print(event["user"]["id"])
print(event["items"][0]["sku"])
print(event.get("coupon"))
print(event.get("coupon", "none"))
print(event["coupon"])
U001
A1
None
none
KeyError: 'coupon'

The coupon key was never sent. .get() gives None or a default you choose, and brackets crash. Across thousands of events, the one that lacks a key is the one that kills the job.

Why does editing a copy change the original?

dict.copy() copies only the top level. Nested dictionaries and lists are still shared. Predict what a shows.

PythonShallow copy versus deep copy
import copy

a = {"user": {"id": "U001"}}
b = a.copy()
b["user"]["id"] = "U999"
print(a["user"]["id"])

c = copy.deepcopy(a)
c["user"]["id"] = "U555"
print(a["user"]["id"])
U999
U999

The shallow copy still points at the same inner dictionary, so editing b changed a. copy.deepcopy() gives a fully independent clone, so editing c left a alone (the second line stays U999).

Watch outdict.copy() is only shallow

dict.copy() duplicates only the top-level dictionary. Nested dictionaries and lists remain shared references. If you edit a nested value in the copy, the original changes too. Use copy.deepcopy() when cloning nested payloads.

Example 3: what happens with duplicate ids?

This mistake gives no error at all. Two rows share an id and we index the batch by id.

PythonTwo rows go in, one comes out
rows = [
    {"id": "ORD-1", "total": 10},
    {"id": "ORD-1", "total": 12},
]

by_id = {row["id"]: row for row in rows}
print(by_id)
print(len(rows), len(by_id))
{'ORD-1': {'id': 'ORD-1', 'total': 12}}
2 1

The later row silently replaced the earlier one. Compare len(rows) with len(by_id) to catch it.

A dict key holds one value. A repeated key quietly overwrites, so check the lengths.

Examples 4 to 6 are optional on a first read. They show up in real pipelines, and you can come back to them after the next lessons.

Example 4: how do you count with Counter?

Counting statuses with a plain dict means checking whether the key exists. collections.Counter counts for you.

PythonCounting category frequencies with Counter
from collections import Counter

statuses = ["paid", "cancelled", "paid", "paid", "refunded", "cancelled"]
counts = Counter(statuses)

print(dict(counts))
print("Most frequent:", counts.most_common(1))
print("Pending count:", counts["pending"])
{'paid': 3, 'cancelled': 2, 'refunded': 1}
Most frequent: [('paid', 3)]
Pending count: 0

A key Counter has never seen returns 0 instead of raising KeyError. most_common(k) returns the top k, ideal for top error codes.

Example 5: how do you group with defaultdict?

defaultdict creates the missing entry for you from a factory such as list or float. No if key not in groups check is needed.

PythonGrouping without key-existence checks
from collections import defaultdict

orders = [
    {"user": "U1", "order_id": "ORD-1", "total": 120.5},
    {"user": "U2", "order_id": "ORD-2", "total": 40.0},
    {"user": "U1", "order_id": "ORD-3", "total": 88.25},
]

by_user = defaultdict(list)
for o in orders:
    by_user[o["user"]].append(o["order_id"])

totals = defaultdict(float)
for o in orders:
    totals[o["user"]] += o["total"]

print(dict(by_user))
print(dict(totals))
{'U1': ['ORD-1', 'ORD-3'], 'U2': ['ORD-2']}
{'U1': 208.75, 'U2': 40.0}

list starts each new key with an empty list and float starts with 0.0.

Example 6: how do you keep a rolling window with deque?

A stream pipeline keeps the latest N values. list.pop(0) is slow because every item shifts left. A deque with maxlen drops the oldest value for free.

PythonA fixed-size rolling buffer with deque
from collections import deque

window = deque(maxlen=3)
stream = [10.0, 20.0, 30.0, 40.0, 50.0]
rolling_averages = []

for val in stream:
    window.append(val)
    avg = sum(window) / len(window)
    rolling_averages.append(round(avg, 1))

print("Final window:", list(window))
print("Rolling averages:", rolling_averages)
Final window: [30.0, 40.0, 50.0]
Rolling averages: [10.0, 15.0, 20.0, 30.0, 40.0]

At 40.0 the oldest value, 10.0, fell out. Memory stays at three items.

Value arrivesWindow holdsAverage
10.010.010.0
20.010.0, 20.015.0
30.010.0, 20.0, 30.020.0
40.020.0, 30.0, 40.030.0
50.030.0, 40.0, 50.040.0
Watch outBrackets that look alike

[ ] makes a list, ( ) makes a tuple, { } with key: value pairs makes a dictionary, and { } with only values makes a set. An empty { } is a dictionary, not a set. To get an empty set write set().

How does the whole lesson replay on ORD-1 to ORD-3?

One batch, four containers
  1. The three orders live in a list, so we can walk them in order.

    1. list of orders
    2. for loop
  2. by_id turns the list into a dictionary, so ORD-3 is one jump away.

    1. list
    2. by_id dict
    3. ORD-3
  3. A set of statuses shows which different values exist: cancelled and paid.

    1. orders
    2. set
    3. 2 statuses
  4. A comprehension keeps the paid totals, 120.5 and 88.25, and sums them to 208.75.

    1. orders
    2. paid_totals
    3. 208.75
  5. If two rows share an id, the dictionary keeps only the last, so compare the lengths.

    1. 2 rows
    2. by_id
    3. 1 key

Four rules to carry forward:

  1. List to walk through, dict to look up, set to deduplicate, tuple for a fixed record.
  2. Use .get() on a dict when a key might be missing, otherwise you get a KeyError.
  3. A comprehension is just a loop that builds a list, written on one line.
  4. A repeated dict key silently overwrites, so compare lengths when you index a batch.

Common beginner questions

Why not store everything in lists?

You can, and for small data it works. But finding an item in a list means checking items one at a time. On a large batch that gets slow, and a dictionary does the same job almost instantly.

What happens if I ask a dictionary for a key that is not there?

With square brackets Python raises a KeyError and stops. With .get(key) you get None, or whatever default you pass as the second argument, and the code carries on.

Is a set the same as a list with duplicates removed?

Close, but a set also throws away the order. If order matters, remove duplicates while you loop and keep a separate set of the ids you have already seen.

What comes next

You can now hold a batch of orders. The next lesson covers decisions and repeats: if, for and while, so you can go through every order and treat each one by its status.

After that comes control flow with error handling, where you skip bad rows, catch an error, and retry a failed call. Then come functions, so you can package this logic and reuse it.

Practice

The exercise gives you the same three orders. First build a dictionary keyed by id, then use a comprehension to collect the totals of the paid orders, print them, and keep the list in result.

Going further: which duplicate wins?

The lesson built dictionaries and sets. A common real job combines them: an events feed contains the same order several times, and you want the latest version of each. A dictionary does this almost by accident, because assigning to an existing key replaces its value. Predict the final status of each order.

PythonKeep the latest event per order
events = [
    {"order_id": "A1", "status": "placed",  "at": 1},
    {"order_id": "A2", "status": "placed",  "at": 2},
    {"order_id": "A1", "status": "shipped", "at": 3},
    {"order_id": "A2", "status": "placed",  "at": 2},      # exact repeat
    {"order_id": "A1", "status": "placed",  "at": 1},      # a late, older copy
]

last_write_wins = {}
for e in events:
    last_write_wins[e["order_id"]] = e["status"]

latest = {}
for e in events:
    current = latest.get(e["order_id"])
    if current is None or e["at"] > current["at"]:
        latest[e["order_id"]] = e

print("last write wins:", last_write_wins)
print("newest timestamp:", {k: v["status"] for k, v in latest.items()})
last write wins: {'A1': 'placed', 'A2': 'placed'}
newest timestamp: {'A1': 'shipped', 'A2': 'placed'}

Order A1 is the trap. 'Last write wins' follows arrival order, so the late old copy overwrites 'shipped' with 'placed'. Comparing timestamps keeps the true latest. The rule you pick is a business decision: arrival order and event time are different things.

RuleGood whenBreaks when
Last one to arrive winsThe feed is always in orderLate or replayed events arrive
Highest timestamp winsEvery event carries a trustworthy time2 events share a timestamp, or clocks are wrong
Highest version number winsThe source assigns increasing versionsThe source does not have versions

Try a new scenario

A list of user ids contains repeats, and you must report how many different users there are and which ids appeared more than once. Which container do you use for each part?

Open the answer

A set answers 'how many different': its size is the count. To find repeats, check membership before adding. The result for this list is 3 ['u1', 'u2']: 3 different users, and u1 and u2 repeated. A set lookup is a single hash check, so this stays fast on millions of ids.

Interview: when would you use a set instead of a list?

When you only need membership tests or uniqueness. Checking whether a value is in a set takes about the same time however large it is, while a list must be scanned from the start. A set loses order and duplicates, so use a list when order or repeats matter.

Close the lesson and explain it

Without scrolling up, explain why a dictionary can keep the wrong version of a record, and what you would compare instead.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Practice this idea

Each page shows the idea moving, then lists problems from easiest. Open one now, or come back when you finish the lesson.

Practice this

Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.

Rate:
Was this useful?