Overview
The shell finds commands on PATH, connects stages with pipes, and treats any non-zero exit as a failed stage. A pipeline is ok only when every exit is 0.
On this page8 sections
What you will do
Nobody watches a nightly pipeline while it runs. A scheduler starts it, and decides whether it worked by looking at one small number.
Two things go wrong again and again. A script that runs on your laptop says command not found on the server. And a pipeline with three steps shows green, even though the middle step crashed.
Both come from how the shell works, so that is what you will learn.
Six words for this lesson.
| Word | What it means |
|---|---|
| PATH | A list of folders the shell searches when you type a command name. |
| Pipe | The | symbol. It sends the output of one command into the next command. |
| Exit code | A number a command returns when it finishes. 0 means it worked. Anything else means it failed. |
| stdout | Where a command sends its normal output. |
| stderr | Where a command sends its error messages. A separate channel. |
| Stage | One step of a pipeline. |
This tab runs Python, not a live shell. The shell commands below are for reading, and for trying later in your own terminal. The practice uses Python to apply the same rule.
Why this skill
Schedulers like Airflow and GitHub Actions never read your logs to decide if a step passed. They read its exit code. If a failure returns 0, the step counts as a success.
PATH: where the shell looks
When you type grep, the shell does not know where grep lives on disk. It looks in each folder listed in PATH, from left to right, and runs the first match.
/usr/local/bin:/usr/bin:/bin- You type grepa name only
- /usr/local/bin?not there
- /usr/bin?found
- Run /usr/bin/grepdone
If no folder has it, the shell prints command not found and returns exit code 127. A program in a folder that is not on PATH is invisible, unless you type its full path, such as /opt/spark/bin/spark-submit.
Python virtual environments use this too. Activating one puts its folder first on PATH, so typing python runs that project's Python.
Here is the same search in Python. We list which folder holds which program, then walk the PATH.
PATH = "/usr/local/bin:/usr/bin:/bin"
installed = {
"/usr/local/bin": ["python3"],
"/usr/bin": ["grep", "gzip", "wc"],
"/bin": ["ls"],
}
def which(name, path):
for folder in path.split(":"):
if name in installed.get(folder, []):
return folder + "/" + name
return None
print(which("grep", PATH))
print(which("spark-submit", PATH))What do you expect for grep? And for spark-submit, which is not installed anywhere in this list?
/usr/bin/grep None
grep is found in the second folder. spark-submit is in none of them, so the answer is None. A real shell would print command not found.
Pipes: the output of one is the input of the next
Here is a small file of orders.
orders.csv, shown as a table. The real file is plain text with commas.
| order_id | status | amount |
|---|---|---|
| 1001 | paid | 40 |
| 1002 | refunded | 15 |
| 1003 | paid | 22 |
How many orders are paid? Use two commands joined by a pipe.
$ grep paid orders.csv | wc -l
2- grep paidkeeps 2 lines
- |pipe
- wc -lcounts them: 2
grep keeps only the paid lines. The pipe hands them straight to wc -l, which counts lines. No temporary file is made in between.
Each program has three channels: stdin to read from, stdout for results and stderr for errors. A pipe joins stdout to the next stdin. Errors skip the pipe and go to your screen, so a failure upstream can be missed by the steps after it.
Exit codes: how a program says it failed
When a program ends, it hands back a number. By a very old rule, 0 means success, and anything else means something went wrong.
Exit codes you will meet in data work.
| Code | Usual meaning | Where you meet it |
|---|---|---|
| 0 | Success | ingest.py finished and wrote its output. |
| 1 | General failure. grep also uses it when nothing matched. | A filter found no records. |
| 2 | Wrong use, such as a bad option or a missing file | grep was given a file that does not exist. |
| 127 | Command not found | python is not on PATH. |
| 137 | Killed, often for using too much memory | A container ran out of memory. |
137 is 128 plus 9, where 9 is the signal that kills a process. In containers it usually means the system stopped your program for using too much memory.
How the code works
A pipeline has several stages, and each returns its own exit code. The whole run worked only if every stage returned 0.
The middle stage exited 1. The chain failed even though gzip returned 0.
Here the filter crashed and exited with 1. gzip then received nothing, compressed nothing, and happily returned 0.
A plain bash pipe reports only the exit code of the last command. So the run looks like a success, although all your records were lost.
Look at the real thing. The file missing.csv does not exist.
$ grep paid missing.csv | wc -l
grep: missing.csv: No such file or directory
0
$ echo $?
0
$ set -o pipefail
$ grep paid missing.csv | wc -l
grep: missing.csv: No such file or directory
0
$ echo $?
2echo $? prints the exit code of the last command. Without pipefail it prints 0, which is wrong, because grep failed. With pipefail it prints 2, the code grep gave.
Production scripts begin with set -euo pipefail. -e stops at the first error, -u treats an unset variable as an error, and -o pipefail makes a pipe fail if any step in it fails.
The rule is easy to write in Python. all() is True only when every item passes.
def pipeline_ok(exit_codes):
return all(code == 0 for code in exit_codes)
print(pipeline_ok([0, 0, 0]))
print(pipeline_ok([0, 0, 1]))True False
One non-zero code is enough to make the answer False.
Worked examples
Two nightly runs, with the exit code of each stage:
Exit codes of two nightly runs.
| Stage | Run A | Run B |
|---|---|---|
| extract | 0 | 0 |
| filter | 0 | 1 |
| compress | 0 | 0 |
Run A is fine. Run B failed, but a plain pass or fail does not say where. A monitor should also name the stage that failed.
def failed_stages(names, codes):
failures = []
for name, code in zip(names, codes):
if code != 0:
failures.append({"stage": name, "exit_code": code})
return failures
names = ["extract", "filter", "compress"]
print(failed_stages(names, [0, 0, 0]))
print(failed_stages(names, [0, 1, 0]))[]
[{'stage': 'filter', 'exit_code': 1}]Run A gives an empty list. Run B points straight at the filter stage and its exit code of 1.
Without set -o pipefail, bash reports only the last command. Put it at the top of every script that runs without a person watching.
One more problem, to see how a missing program looks. nosuchcmd is not a real command:
$ nosuchcmd
bash: nosuchcmd: command not found
$ echo $?
127On a Mac, wc -l prints its number with extra spaces in front. That is only formatting.
Common beginner questions
Why does my script work on my laptop but say command not found on the server?
The program is installed in a folder that is on your laptop's PATH but not on the server's. Install it on the server, or use its full path.
Does a pipe wait for the first command to finish?
No. The commands run at the same time, and data flows through as it is made. That is why a pipe can handle files bigger than memory.
Why does grep return 1 when it finds nothing?
By convention, 1 means no lines matched. It is not a crash, but a script with set -e will still stop on it. That surprises many people.
Schedulers decide pass or fail from one number: the exit code. 0 is success.
- exit code
- 0 = success
The shell finds a command by searching the folders in PATH. No match gives 127.
- PATH
- found or 127
A pipe sends the output of one command into the next.
- grep
- |
- wc -l
A plain pipe reports only the last stage. A failed stage upstream can hide.
- stage fails
- last stage 0
- looks green
set -o pipefail makes the whole pipe fail if any stage fails.
- pipefail
- exit 2
pipeline_ok is True only when every exit code is 0.
- all codes 0
- True
What comes next
Next you will run your first Docker container. It packs a program and what it needs into a box, so it runs the same on every machine.
Practice
Run the sample code to see how pipeline_ok treats a list of exit codes.
Write pipeline_ok(exit_codes) to return True only when every number in the list is 0. Call it on [0, 0, 1], save the answer in result, and print it. You should get False.
Going further: chaining commands and reading what failed
Every command ends with an exit code: 0 means success, anything else means failure. && and || use it to decide what runs next, and a pipeline can use it too. Predict which of the echo lines run.
false; echo "exit code: $?"
true && echo "ran because the first succeeded"
false && echo "never printed"
false || echo "ran because the first failed"
ls missing_folder 2> errors.txt; echo "ls exit code: $?"
cat errors.txt
printf 'a.csv\nb.csv\n' | xargs -I{} echo processing {}
export STAGE=dev
bash -c 'echo "child sees STAGE=$STAGE"'exit code: 1 ran because the first succeeded ran because the first failed ls exit code: 1 ls: missing_folder: No such file or directory processing a.csv processing b.csv child sees STAGE=dev
false returned a non-zero code, so && skipped its right side and || ran its right side. The failed ls wrote its message to errors.txt because of 2>, and its exit code (1 on this system, 2 on some others) is still available. xargs turned each input line into one command.
And export made STAGE visible to the child process, which a plain assignment would not.
| Pattern | Meaning | Use |
|---|---|---|
| a && b | Run b only if a succeeded | Steps that depend on each other |
| a || b | Run b only if a failed | A fallback or an alert |
| a; b | Run b either way | Independent steps |
| export NAME=value | Pass a variable to child programs | Configuration for scripts |
The exact wording of error messages and the number a failing command returns vary between systems, but the rule is stable: zero is success.
In a script, set -e stops at the first failure, and set -o pipefail makes a pipeline fail if any stage fails, so a broken first stage cannot hide behind a successful last one.
Try a new scenario
A nightly script runs: download_data; transform_data; publish_report. The download fails but the report is still published. What is wrong and how do you fix it?
Work it out, then open the answer
The semicolons run every step regardless of earlier failures, so transform and publish used stale or missing data. Chain the steps with && so each one runs only if the previous one succeeded, or start the script with set -e so it stops at the first failure. Then a failed download leaves yesterday's report in place instead of publishing a wrong one.
What does set -e do in a script?
It makes the script stop at the first command that fails, so later steps do not run on broken input. Pair it with pipefail so a failure inside a pipeline is not hidden.
Close the lesson and explain it
Without scrolling up, explain the difference between && and ;, and what a non-zero exit code means.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.