Overview
The shell finds commands on PATH, connects stages with pipes, and treats any non-zero exit as a failed stage. A pipeline is ok only when every exit is 0.
On this page7 sections
What you will do
When you type a command name in a terminal, the shell needs to find the program file for that command. PATH is a colon-separated list of directories the shell searches, in order, for the program you named. If python is not in any directory on PATH, the shell cannot start ingest.py. It exits with code 127 (command not found) before your script ever runs.
A pipe connects the output of one program to the input of the next. The command grep paid orders.csv | wc -l sends the matching lines from grep into wc, which counts them. Each program in the pipe has its own exit code. Convention: 0 means success, any other integer means failure.
Exit codes and pipes are fundamental to automation. Every CI step, every cron job, and every deployment script relies on checking whether the previous command succeeded before running the next one.
Why this skill
Imagine your team runs a nightly pipeline: extract data, filter rows, compress the output. If the filter stage finds zero matching rows and exits with code 1, the pipeline should stop and alert you. Without checking exit codes, the next stage (gzip) happily compresses an empty file, and nobody notices until a dashboard goes blank the next morning.
A pipe connects stdout of one process to stdin of the next. Each process still has its own exit code. Convention: 0 means success. Any other integer means that stage failed (1 often means no match for grep, 2 a usage error, 137 a kill signal, typically from running out of memory).
Understanding PATH, pipes, and exit codes is not optional. You will see them in Dockerfiles (RUN commands), CI workflows (step failures), and Kubernetes logs (container exit codes).
How the code works
pipeline_ok is true only when every collected exit code is 0. One 1 in a three-stage pipe means the night job should stop, not quietly gzip an empty stream. This tab runs Python, not bash. The function is the contract you will later put in a CI step.
Mental model: where the command lives, how stages connect, what zero means
PATH is a colon-separated list of directories. The shell searches those directories, left to right, for the name you typed. A missing python is not a Python exception. It is the shell saying command not found, usually exit 127.
A pipe connects stdout of one process to stdin of the next. Each process keeps its own exit code. grep returning 1 (no match) is a failed stage if you required a match. pipeline_ok is true only when every collected code is 0.
A 1 in the middle is a failed pipeline even if gzip exits 0.
Exit codes you will actually see
Collect every stage. Do not trust the last process alone.
| Code | Usual meaning | Warehouse example |
|---|---|---|
| 0 | Success | ingest wrote bronze |
| 1 | General failure / grep no match | Filter found zero paid rows |
| 2 | Misuse (bad flags) | gzip given a directory by accident |
| 127 | Command not found | python not on PATH |
| 137 | Killed (often OOM) | The 12 GB log ate the worker |
Worked example: all zeros, or False
You write pipeline_ok(exit_codes) using Python's all() function. [0, 0, 0] returns True. [0, 0, 1] returns False, and that False is what you print. If you only checked the last code (gzip's 0), you would miss the filter failure.
def pipeline_ok(exit_codes):
return all(code == 0 for code in exit_codes)
result = pipeline_ok([0, 0, 1])
print(result)Diagnosing which stage failed
When a pipeline fails, you need to know which stage caused the problem. This function returns the name and exit code of every failing stage.
def failed_stages(names, codes):
failures = []
for name, code in zip(names, codes):
if code != 0:
failures.append({"stage": name, "exit_code": code})
return failures
stages = ["extract", "filter", "compress"]
codes = [0, 1, 0]
print(failed_stages(stages, codes))PATH in practice
When a container or CI runner reports command not found, the fix is either to install the program or to add its directory to PATH. This function simulates a PATH lookup.
def find_command(command, path_dirs):
for directory in path_dirs:
if command in directory:
return directory
return None
path = ["/usr/bin", "/usr/local/bin", "/home/app/.local/bin"]
print(find_command("python3", ["/usr/bin/python3"]))
print(find_command("spark-submit", path))pipefail in real bash
Without set -o pipefail, bash reports only the exit code of the last command in a pipe. Production scripts use set -euo pipefail to catch mid-pipe failures. The Python function here captures the same idea.
Exit code 137 means the process was killed
If your container exits 137, it usually ran out of memory and was killed by the OS. Increasing memory limits or optimizing the code are the fixes, not retrying.
Functions and all() from Core Python
You practiced writing functions and using all() with generator expressions in Core Python. pipeline_ok is the same pattern: a function that returns a boolean based on checking every item in a list.
Wrong and right
The wrong check looks only at the last stage's exit code. The right check collects every stage. PATH problems show up as 127 before your code even runs; logging "ingest started" never happens. Fix PATH or call /usr/bin/python3 explicitly.
- PATH is where the shell finds command names.
- Pipes connect stdout to stdin. Exit codes stay per process.
- pipeline_ok requires every code to be 0.
Worked examples
A three-stage pipe collects exit codes from extract, filter, and compress. pipeline_ok is True only when every code is 0.
Input: per-stage exit codes from a nightly job.
| stage | exit code (sample A) | exit code (sample B) |
|---|---|---|
| extract | 0 | 0 |
| filter | 0 | 1 |
| compress | 0 | 0 |
def pipeline_ok(exit_codes):
return all(code == 0 for code in exit_codes)
print(pipeline_ok([0, 0, 0]))
print(pipeline_ok([0, 1, 0]))
result = pipeline_ok([0, 0, 1])[0, 0, 0] prints True. [0, 1, 0] prints False even though gzip (the last stage) exited 0. Checking only the last code would miss the filter failure.
Output: collect every stage, not just the last process in the pipe.
| exit_codes | pipeline_ok | which stage failed |
|---|---|---|
| [0, 0, 0] | True | none |
| [0, 1, 0] | False | filter (middle) |
Copy-paste without reading the output
Run Sample first. If the numbers or row count look wrong, stop and re-read the previous section before changing code.
Common beginner questions
Why not just run scripts on my laptop?
Your laptop works differently from a production server. Different Python version, different OS, different installed packages. Code that works locally can fail in production. Infrastructure tools (Docker, CI) create a consistent environment.
What is stdout?
stdout (standard output) is the default destination for a program's printed output. When you pipe commands, stdout of one program becomes the input (stdin) of the next.
Do I need to memorize exit codes?
No. Remember that 0 means success and anything else means failure. Look up specific codes when you encounter them. 127 (not found) and 137 (killed) are the two worth recognizing immediately.
What comes next
The next lesson introduces Dockerfiles. Docker packages your code, dependencies, and environment into a single image. Exit codes matter inside Docker containers too: a non-zero exit from your CMD means the container failed.
Practice
Run Sample and compare the all-green print with the grep-missed print. Then complete Exercise: implement pipeline_ok, call it on [0, 0, 1], store False in result, and print it.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.