DE data structures

Do not start by asking which collection is "best." Start by asking what operation your pipeline needs to perform repeatedly. If the question is "process every row in arrival order," a list is natural. If the question is "find the row for this ID," a dictionary is natural. If the question is "have I seen this value before?" a set is often natural. If the question is "represent a fixed combination of values," a tuple may fit.

The performance difference is not just a computer-science detail. Data pipelines repeatedly perform the same operations over thousands or millions of records. A structure that matches the operation can remove entire loops from your program. That improves both runtime and readability because the code expresses the question directly.

Also separate two ideas that beginners often mix up: changing a container and changing the objects inside it. A new list can still contain references to the same dictionaries as the old list. This is why copying a container is not always the same as copying all nested data. You will meet this distinction again when pipeline stages share records.

Python provides four built-in ways to organize collections of data: lists, tuples, dictionaries, and sets. Each one is designed for a different job. Choosing the right container makes your code faster, clearer, and less error-prone.

A list is an ordered collection that can grow, shrink, and change. A tuple is like a list that is locked after creation (immutable). A dictionary (dict) maps unique keys to values, like a phone book mapping names to numbers. A set is an unordered collection that automatically removes duplicates.

A list comprehension is a concise way to create a new list by transforming or filtering an existing collection. The syntax [expression for item in collection if condition] replaces several lines of loop code with one readable line.

Read a comprehension left to right once you know the pieces. The for item in collection part is the same loop you already write. The expression before for is what each item becomes. The optional if at the end is the filter. Start with a plain for loop if a comprehension feels opaque; both produce the same list, and clarity beats cleverness.

Imagine a payment processor sends your company 50,000 order records every hour. You need to look up individual orders by their ID, detect duplicate submissions, filter for paid orders, and compute totals. If you store all orders in a plain list and search it every time, each lookup scans up to 50,000 items. With a dictionary keyed by order_id, each lookup is nearly instant regardless of how many orders exist. Choosing the wrong data structure can turn a 1-second job into a 10-minute job.

Think of a list like a numbered row of lockers: items have positions (index 0, 1, 2...) and you can add or remove lockers. A dictionary is like a filing cabinet with labeled folders: you find a folder instantly by its label without checking every drawer.

Here is sample order data we will organize into different structures:

Run the example below in this tab. Read the input, follow the code, then check the output matches what you expect.

The list comprehension collects only the totals from paid orders. The dictionary comprehension builds an index so you can find any order by its ID instantly. The set comprehension collects all unique statuses, automatically removing duplicates.

One beginner trap is that a variable does not contain a separate copy of a list or dictionary; it points to an object. If two names point to the same list, changing it through either name changes the same batch. This matters when a transform is meant to leave its input untouched: make a copy or build a new list rather than accidentally changing data that another stage still needs.

The second example uses {**row, ...} to create a new dictionary for each row. That style is often safer in a pipeline: raw input remains available for auditing, while the next stage receives a clean result. In-place changes with row["status"] = ... are not wrong, but use them deliberately and document which stage owns the data.

Dictionaries cannot have duplicate keys. If two orders share the same ID, the second silently overwrites the first. Always verify uniqueness before building a dictionary index from batch data.

The wrong approach scans the entire list for every lookup. With 50,000 orders and 1,000 lookups, that is 50 million comparisons. The right approach builds the dictionary once (50,000 operations) and then each lookup is instant. For large datasets, this difference is enormous.

A tuple's other job, beyond freezing a record, is standing in as a compound key: a dictionary key made of more than one value glued together.

Revenue per product per day is a common report, and neither date alone nor product alone is a unique key for it. The pair (date, product) is. A list could not do this job even if you wanted it to: dictionary keys must be hashable, meaning Python can compute a fixed fingerprint for them, and only immutable values like tuples, strings, and numbers qualify. A list can change after creation, so its fingerprint could not stay valid, and Python refuses it as a key.

CPython dictionaries and sets are implemented as compact open-addressed hash tables with a sparse index array and a dense entries array. When you query `key in dictionary`, Python hashes the key using `hash(key)`, maps it to an index modulo the table size, and checks the slot. In the absence of hash collisions, lookup is O(1). If two keys produce the same hash slot, Python probes alternate slots using a perturbation formula. Sets and dicts automatically resize when they reach two-thirds capacity (a load factor of 0.66).

In daily banking reconciliation (such as matching PhonePe or Google Pay transaction logs against a bank core ledger), a pipeline receives 2 million records from the app gateway and 2 million records from the bank. If you use a list lookup (`for row in gateway: if row['tx_id'] in bank_list:`), searching a 2-million item list for each record triggers $2,000,000 \times 2,000,000 = 4 \times 10^{12}$ comparisons, freezing the pipeline for days. By indexing bank transaction IDs into a Python set (`bank_tx_set = {row['tx_id'] for row in bank_records}`), the membership check executes in O(1) time. The entire 2-million record reconciliation completes in under 2 seconds.

A classic junior engineer trap occurs when attempting to copy records: `working_batch = raw_records.copy()` or `list(raw_records)`. This creates a shallow copy. The outer list is fresh, but every dictionary inside the list points to the exact same memory address as the original. Modifying `working_batch[0]['status'] = 'processed'` silently mutates `raw_records[0]['status']` as well, destroying the raw source audit trail. When you need independent dictionaries, use dictionary comprehension copies (`[{**row} for row in raw_records]`) or `copy.deepcopy()`.

An interviewer asks: 'What is the time complexity of searching for an item in a Python list versus a set, and why? Under what circumstances can a set lookup degrade?' A strong answer: List lookup is O(N) because it performs a linear sequential scan from index 0 to N. Set lookup is O(1) average time because it hashes the key and directly inspects the hash bucket. It can degrade to O(N) only in the pathological scenario where every key has the exact same hash collision, forcing Python to linearly probe every entry.

You now know how to organize data into the right containers. The next lesson covers control flow: making decisions with if/else, repeating work with loops, and handling errors gracefully.

Run Sample to see the dictionary lookup, unique statuses, and paid totals. Then complete the Exercise: build a dictionary keyed by id, collect paid totals into a list called result that equals [120.5, 88.25], and print it.