Lesson 1 of 5 in this module

7/43 in track

Control flow & error handling

python10 min

Overview

Real jobs meet bad rows and temporary failures. Learn how to choose a path, catch expected errors, and retry without hiding the real problem.

On this page8 sections

What you will do

Real input is messy. One order has an empty total, another has a status you have never seen, and an API call fails once and works on the second try.

The last lesson showed if and for. This one adds the tools that keep a job running through the mess: try and except for errors, while for waiting, break and continue for skipping, and a retry loop that joins them.

What happens when Python hits an error?

By default it stops the whole program at that line. Nothing below it runs. Predict which lines print here.

PythonOne bad value
print(float("12.5"))
print(float("cheap"))
print("done")
12.5
Traceback (most recent call last):
  File "main.py", line 2, in <module>
    print(float("cheap"))
          ~~~~~^^^^^^^^^
ValueError: could not convert string to float: 'cheap'

The first line worked. The second raised a ValueError, which is Python's name for a value of the right kind but with unusable content. The third line, done, never ran.

For one value on your own laptop that is fine. For 100,000 rows in a nightly job, one bad row must not kill the other 99,999.

How do try and except keep the job running?

Put the risky lines under try. If one of them raises the error you named, Python jumps to the except block, runs it, and then carries on after the whole try and except. Predict the four lines.

PythonA bad price is reported, not fatal
raw_prices = ["12.5", "cheap", "9"]

for raw in raw_prices:
    try:
        price = float(raw)
        print("price is", price)
    except ValueError:
        print("bad price:", raw)

print("done")
price is 12.5
bad price: cheap
price is 9.0
done

For cheap, float raised the error, so print("price is", price) was skipped and the except line ran instead. The loop then moved on to 9, and done printed at the end.

Inside try, float(raw) either works and the next line runs, or raises a ValueError and Python jumps to the except block. In both cases the loop continues with the next price.
try is the plan A path. except is the path Python takes only when the named error appears. Either way, the program keeps going.

try and except turn a crash into a decision. You choose what to do with the bad row.

Which tool do I use when?

You met if and for in the last lesson. The table adds the rest, and the lesson walks through each row.

Choose the tool from the question you are asking: which branch, how many times, or what if it fails.

ToolWhen to use itHow it stopsPipeline example
if / elif / elseDifferent actions per conditionOne branch executesProcess paid orders versus cancelled ones
for loopKnown collections or fixed attemptsCollection exhausted or breakIterate order records in a daily batch
while loopRepeat until a condition changesCondition turns False or breakPoll an export status until complete
try / exceptAn operation might raise an errorMatching except block runsCatch a network timeout on an API call
finallyCleanup that must always happenRuns after try and exceptClose an open network socket or file handle

Three loop helpers refine a loop. break leaves the loop at once, continue skips to the next item, and the else on a for loop runs only when no break happened.

Working with order records

We use 4 sample orders across the rest of this lesson. Each row has a status and an amount.

Different statuses need different handling.

order_idpriceqtystatus
ORD-119.504paid
ORD-28.002cancelled
ORD-312.252paid
ORD-49.001paid

Python marks a block with indentation, which means the spaces at the start of a line. Everything under if, for, while, try or def is indented by four spaces. Uneven spacing raises an IndentationError before the program starts.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
ConditionA test whose answer is True or False
if / elsePick which lines run based on a condition
LoopRepeat the same steps for each item
whileA loop that repeats until a condition turns False
breakLeave a loop early
RetryTry the same step again after a failure

Why this skill

Pipelines process variable input data where network connections drop and source schemas drift unexpectedly.

Guarantees of loop termination

When reading or writing loops, verify three structural guarantees:

  1. What variables change on each iteration?
  2. What condition determines the branch path?
  3. What guarantees that the loop will terminate?

Infinite loops consume CPU resources endlessly while delaying downstream transformation jobs.

Exceptions carry diagnostic details about failed operations. Catch them only when your code can recover or log a meaningful alert.

The cost of transient failures

Production pipelines frequently make network requests to external APIs that intermittently return 503 errors.

  1. 503 timeoutno retry loop
  2. Job crashespartial batch left
  1. Reports stallmissing morning metrics

Bounded retries attempt the request up to 3 times before failing. This tolerates brief network glitches while surfacing permanent outages.

Retry transient failures a fixed number of times. Never retry indefinitely and never swallow the error.

How the code works

Python evaluates conditions in sequence, executes matched blocks, and handles raised exceptions.

Branching on order status

Loop over the orders and let each row status pick its execution path.

PythonBranching on order status
orders = [
    {"order_id": "ORD-1", "price": 19.50, "qty": 4, "status": "paid"},
    {"order_id": "ORD-2", "price": 8.00, "qty": 2, "status": "cancelled"},
    {"order_id": "ORD-3", "price": 12.25, "qty": 2, "status": "paid"},
]

paid_total = 0.0
for row in orders:
    if row["status"] == "paid":
        subtotal = row["price"] * row["qty"]
        paid_total += subtotal
        print(f"  {row['order_id']}: subtotal {subtotal}")
    elif row["status"] == "cancelled":
        print(f"  {row['order_id']}: skipped (cancelled)")
    else:
        raise ValueError(f"Unknown status: {row['status']}")

print("Paid total:", paid_total)

Predict the paid total before revealing the output below:

  ORD-1: subtotal 78.0
  ORD-2: skipped (cancelled)
  ORD-3: subtotal 24.5
Paid total: 102.5

Orders ORD-1 and ORD-3 updated the total; ORD-2 was skipped.

order_idBranch executedAction taken
ORD-1if status == paidAdded 78.0 to paid_total
ORD-2elif status == cancelledSkipped computation
ORD-3if status == paidAdded 24.5 to paid_total

Place the most common branch first so the majority of incoming records evaluate on the initial comparison.

The trap with and and or operator precedence

A common logic mistake in conditional checks is assuming that comparisons evaluate strictly left to right. In Python, and binds tighter than or.

PythonOperator precedence trap with and versus or
country = "IN"
total = 10

# Without parentheses: and binds tighter than or
cond1 = country == "IN" or country == "US" and total > 100
# With parentheses: explicit grouped logic
cond2 = (country == "IN" or country == "US") and total > 100

print("Without parentheses:", cond1)
print("With parentheses:", cond2)

Predict what prints for cond1 when total is only 10: does it evaluate to True or False?

Without parentheses: True
With parentheses: False

Because and has higher precedence, Python grouped country == 'US' and total > 100 first. The condition evaluated to True because country was 'IN'.

Always wrap combined or expressions in explicit parentheses when filtering records.

Defensive else for unexpected values

The else branch handles records that match neither paid nor cancelled. Raising an exception stops the job with a clear error, which your monitoring can turn into an alert.

PythonCatching unhandled statuses
orders = [
    {"order_id": "ORD-1", "status": "paid"},
    {"order_id": "ORD-5", "status": "refunded"},
]

for row in orders:
    if row["status"] == "paid":
        print(row["order_id"], "paid")
    elif row["status"] == "cancelled":
        print(row["order_id"], "skipped")
    else:
        raise ValueError(f"Unknown status: {row['status']}")
ORD-1 paid
Traceback (most recent call last):
  File "pipeline.py", line 11, in <module>
    raise ValueError(f"Unknown status: {row['status']}")
ValueError: Unknown status: refunded

When a payment gateway introduces a new refunded status, the pipeline stops on that record instead of dropping revenue silently.

A visible exception fixed in minutes is far better than silent data corruption found during an audit.

Bounded retries with loop break

A retry loop runs up to 3 attempts. When an attempt succeeds, break exits the loop. If all attempts fail, for/else executes.

Bounded 3-attempt retry sequence
1attempt 12RuntimeError3attempt 24RuntimeError5attempt 36success -> break

Failed attempts catch RuntimeError and retry. Success triggers break immediately.

PythonBounded retry with break on success
def flaky_api(attempt: int) -> str:
    """Simulates an API that fails twice before succeeding."""
    if attempt < 3:
        raise RuntimeError("API returned 503")
    return "ok"

result = None
for attempt in range(1, 4):
    try:
        result = flaky_api(attempt)
        print(f"  Attempt {attempt}: SUCCESS")
        break
    except RuntimeError as exc:
        print(f"  Attempt {attempt}: {exc}")
else:
    raise RuntimeError("Failed after 3 attempts")

print("Result:", result)

Predict which attempt succeeds and what prints after the loop exits:

  Attempt 1: API returned 503
  Attempt 2: API returned 503
  Attempt 3: SUCCESS
Result: ok

Attempts 1 and 2 raised RuntimeError and were caught. Attempt 3 succeeded, assigned result, and executed break.

Because break exited the loop on attempt 3, the for loop else block was skipped.

RememberCap every retry loop

Always set a fixed upper limit like range(1, 4) on retry loops. Retrying without a limit turns temporary network hiccups into permanent hangs.

Traps in poorly constructed retries

Compare an unsafe infinite retry against a bounded retry that catches specific exceptions.

Safe retries combine fixed iteration counts, specific exception classes, and terminal alerts.

Unsafe patternWhy it fails in productionSafe replacement
while True with no counterLoops forever on permanent errorsfor attempt in range(1, 4)
except Exception:Hides coding bugs and syntax typosexcept (RuntimeError, ConnectionError):
except: passSwallows errors and pretends work succeededLog error, record attempts, then re-raise

While loops and termination conditions

A while loop repeats as long as its condition evaluates to True. Every while loop requires an internal state advancement.

PythonPolling until status becomes done
checks = ["running", "running", "done"]
i = 0
status = checks[i]
while status != "done":
    print("still", status)
    i += 1
    status = checks[i]
print("finished after", i, "checks")

Predict the number of checks printed before finished:

still running
still running
finished after 2 checks

On each turn, i increments by 1, advancing the status pointer. When status reaches done, the loop condition turns False.

Guaranteed cleanup with finally

A finally block executes whether the try block completes, catches an error, or returns early. Use it for resource cleanup.

Pythonfinally runs on both success and failure
def parse_price(raw: str):
    try:
        return float(raw)
    except ValueError:
        print("bad price:", raw)
        return None
    finally:
        print("cleanup check for", raw)

print(parse_price("12.5"))
print(parse_price("cheap"))

Predict the order of printed output for the invalid string cheap:

cleanup check for 12.5
12.5
bad price: cheap
cleanup check for cheap
None

Notice that cleanup printed before returning 12.5 for the valid row, and after the error message for cheap.

In production pipelines, finally guarantees that resources such as open database connections, temporary files, and network sockets are closed properly even when an exception interrupts earlier statements.

Differences between break and continue

continue bypasses remaining lines for the current record. break halts the entire loop immediately upon matching.

Pythoncontinue skips 1 row while break ends the loop
orders = [
    {"order_id": "ORD-1", "price": 19.50, "qty": 4, "status": "paid"},
    {"order_id": "ORD-2", "price": 8.00, "qty": 2, "status": "cancelled"},
    {"order_id": "ORD-3", "price": 12.25, "qty": 2, "status": "paid"},
    {"order_id": "ORD-4", "price": 9.00, "qty": 1, "status": "paid"},
]

first_small = None
for row in orders:
    if row["status"] == "cancelled":
        continue
    if row["price"] * row["qty"] < 30:
        first_small = row["order_id"]
        break

print("First small paid order:", first_small)
First small paid order: ORD-3

break stops on the first match, leaving later orders unread.

Order seenLoop actionReason
ORD-1Evaluated conditionSubtotal 78.0 is not under 30
ORD-2continue executedCancelled status bypassed immediately
ORD-3break executedSubtotal 24.5 matched condition; loop halted
ORD-4Never reachedLoop stopped at ORD-3

Worked examples

Apply these control flow patterns to real data engineering scenarios.

Catching multiple distinct error types

In ingest pipelines, a single raw dictionary can fail in multiple distinct ways. A column may be missing, or its string value may fail casting.

PythonCatching KeyError versus ValueError independently
def parse_order(row: dict) -> str:
    try:
        subtotal = float(row["price"]) * int(row["qty"])
        return f"subtotal: {subtotal}"
    except KeyError as err:
        return f"Missing column: {err}"
    except ValueError as err:
        return f"Malformed number: {err}"

print(parse_order({"price": "19.50", "qty": "4"}))
print(parse_order({"price": "19.50"}))
print(parse_order({"price": "invalid", "qty": "4"}))

Predict what prints for the row missing the qty key:

subtotal: 78.0
Missing column: 'qty'
Malformed number: could not convert string to float: 'invalid'

Each except block handles its specific failure mode appropriately without conflating missing schema fields with corrupted numerical values.

Idempotent processing using set lookups

When a nightly pipeline re-runs, it must avoid double-processing records that were already committed.

PythonSkipping previously settled orders
settled_ids = {"ORD-1"}
processed = []

incoming = [
    {"order_id": "ORD-1"},
    {"order_id": "ORD-3"},
]

for row in incoming:
    if row["order_id"] in settled_ids:
        print(row["order_id"], "already processed, skipping")
        continue
    settled_ids.add(row["order_id"])
    processed.append(row["order_id"])
    print(row["order_id"], "processed now")

print("Processed IDs:", processed)
ORD-1 already processed, skipping
ORD-3 processed now
Processed IDs: ['ORD-3']

Re-running the batch skips ORD-1 without creating duplicate transactions, processing only newly arrived records.

Isolating bad rows into quarantine

Instead of letting one invalid string halt a 100,000-row batch, partition records into clean and quarantine lists.

PythonSeparating valid amounts from bad rows
raw_records = ["12.50", "corrupted", "88.25", "invalid", "15.00"]
clean_values = []
quarantined = []

for item in raw_records:
    try:
        clean_values.append(float(item))
    except ValueError:
        quarantined.append(item)

print("Clean:", clean_values)
print("Quarantined:", quarantined)
Clean: [12.5, 88.25, 15.0]
Quarantined: ['corrupted', 'invalid']

Valid records continue into analytics, while unparseable strings are preserved in quarantine for team inspection.

Silent data loss from bare except

Wrapping record parsing in a bare except block hides bugs and results in silent data loss.

PythonSilent failure hiding a column rename
rows = [{"user_id": "U1"}, {"user_id": "U2"}]
loaded = 0

for row in rows:
    try:
        cid = row["customer_id"]
        loaded += 1
    except:
        pass

print("Loaded count:", loaded)
Loaded count: 0

When the upstream source renamed customer_id to user_id, every row raised KeyError.

The bare except swallowed every error, leading the job to report success while loading zero records.

Watch outNever use bare except

Writing except: or except Exception: catches typos and unexpected bugs along with expected failures. Always name the specific exception you intend to handle.

Replay on 3 orders

From conditional branches to bounded retries
  1. The for loop processes ORD-1, ORD-2, and ORD-3 sequentially.

    1. orders
    2. for row
  2. if/elif routes records: paid adds subtotal, cancelled skips, unexpected raises.

    1. row
    2. branch
    3. total 102.5
  3. A flaky call raises RuntimeError twice, and bounded retries catch both attempts.

    1. fetch
    2. RuntimeError
    3. retry
  4. Attempt 3 succeeds and break exits the loop; for/else raises only if all attempts fail.

    1. attempt 3
    2. break
    3. result ok
  5. Catching only RuntimeError ensures unexpected programming bugs surface immediately.

    1. except RuntimeError
    2. bugs surface

Four operational rules for pipeline control flow:

  1. Identify the variable that changes on each turn, the branch condition, and the termination guarantee.
  2. Raise an explicit exception in the else branch to detect upstream schema drift immediately.
  3. Bound every retry loop with a fixed iteration count and catch only expected exception classes.
  4. Use break to halt on first match, continue to skip invalid rows, and never use bare except.

Common beginner questions

When do I use for versus while?

Use for when iterating over a known collection, a range of attempts, or a fixed dataset. Use while when waiting for an external condition to change, such as polling an export job until its status turns complete. Every while loop must advance its own termination condition.

What is the difference between break and continue?

break terminates the entire loop immediately and skips any remaining items. continue aborts only the current iteration and jumps directly to the next item in the collection.

What does the else clause on a for loop do?

The else block runs only if the loop completed all iterations without encountering a break statement. In retry loops, hitting break indicates success, so the else block provides the natural location to raise a terminal error after all attempts fail.

Why is except Exception considered risky in data pipelines?

except Exception catches almost all runtime errors, including misspelled variable names, missing keys, and TypeErrors. Catching it blindly conceals real defects and makes debugging difficult. Always specify the exact exception you expect to recover from.

What comes next

You now control execution paths, handle runtime errors, and write bounded retry loops. Next, you will package pipeline transformations into reusable, testable functions and modules.

Practice

In the exercise, wrap flaky() in a retry loop allowing at most 3 attempts. Catch RuntimeError specifically, print SUCCESS when an attempt succeeds, and store 'ok' in result.

Going further: what does an if actually test?

An if does not only compare numbers. Python turns any value into True or False, and a handful of values count as False: 0, an empty string, an empty list, an empty dictionary and None. That is useful, and it hides a trap when the number 0 is a real answer.

The program below checks 8 values with a bare if. Predict which ones take the if branch before you open the output.

PythonWhich values count as True?
values = [0, 1, "", "paid", [], [1], {}, None]
for v in values:
    if v:
        print(repr(v), "-> truthy")
    else:
        print(repr(v), "-> falsy")
0 -> falsy
1 -> truthy
'' -> falsy
'paid' -> truthy
[] -> falsy
[1] -> truthy
{} -> falsy
None -> falsy

Falsy values. Everything else is truthy.

ValueCounts asTypical meaning in a pipeline
0 or 0.0FalseA real zero, such as 0 returns
"" (empty text)FalseA field that arrived blank
[] or {}FalseNo rows or no keys
NoneFalseNo value at all
Any other valueTrueData is present

The trap is a field where 0 is valid. A refund amount of 0 and a missing refund amount are different facts, yet a bare if treats them the same. Predict what the two checks below print for a refund of 0.

PythonA refund of 0
refund = 0

if refund:
    print("bare if: refund recorded")
else:
    print("bare if: no refund")

if refund is not None:
    print("is not None: refund recorded")
else:
    print("is not None: no refund")
bare if: no refund
is not None: refund recorded

Try a new scenario

A pipeline reads rows[0] only if rows is not empty. A teammate writes if rows == None. For an empty list, does that check protect the code, and what would you write instead?

Work it out, then open the answer

It does not protect it. An empty list is not None, so rows == None is False and the code goes on to read rows[0], which raises an IndexError. Write if rows: (or if len(rows) > 0:) to test for emptiness. Use is None only when None is the specific value you want to detect.

Why is is None preferred over == None?

None exists only once in a Python program, so is None asks the exact question: is this the None value. It also cannot be fooled by an object that defines its own equality. For a beginner the practical rule is: use is None and is not None when you mean missing, and a bare if when you mean empty or zero.

Close the lesson and explain it

Without scrolling up, name 3 values that count as False and explain why a bare if is risky for a numeric field where 0 is a valid answer.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Practice this

Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.

Rate:
Was this useful?