Lesson 5 of 6 in this module

5/13 in track

PATH, pipes, and exit codes

python12 min

Overview

The shell finds commands on PATH, connects stages with pipes, and treats any non-zero exit as a failed stage. A pipeline is ok only when every exit is 0.

On this page8 sections

What you will do

Nobody watches a nightly pipeline while it runs. A scheduler starts it, and decides whether it worked by looking at one small number.

Two things go wrong again and again. A script that runs on your laptop says command not found on the server. And a pipeline with three steps shows green, even though the middle step crashed.

Both come from how the shell works, so that is what you will learn.

Six words for this lesson.

WordWhat it means
PATHA list of folders the shell searches when you type a command name.
PipeThe | symbol. It sends the output of one command into the next command.
Exit codeA number a command returns when it finishes. 0 means it worked. Anything else means it failed.
stdoutWhere a command sends its normal output.
stderrWhere a command sends its error messages. A separate channel.
StageOne step of a pipeline.
RememberWhat this tab runs

This tab runs Python, not a live shell. The shell commands below are for reading, and for trying later in your own terminal. The practice uses Python to apply the same rule.

Why this skill

Schedulers like Airflow and GitHub Actions never read your logs to decide if a step passed. They read its exit code. If a failure returns 0, the step counts as a success.

PATH: where the shell looks

When you type grep, the shell does not know where grep lives on disk. It looks in each folder listed in PATH, from left to right, and runs the first match.

ShellA typical PATH. Folders are separated by colons.
/usr/local/bin:/usr/bin:/bin
  1. You type grepa name only
  2. /usr/local/bin?not there
  3. /usr/bin?found
  1. Run /usr/bin/grepdone

If no folder has it, the shell prints command not found and returns exit code 127. A program in a folder that is not on PATH is invisible, unless you type its full path, such as /opt/spark/bin/spark-submit.

Python virtual environments use this too. Activating one puts its folder first on PATH, so typing python runs that project's Python.

Here is the same search in Python. We list which folder holds which program, then walk the PATH.

PythonSearch PATH for a program
PATH = "/usr/local/bin:/usr/bin:/bin"
installed = {
    "/usr/local/bin": ["python3"],
    "/usr/bin": ["grep", "gzip", "wc"],
    "/bin": ["ls"],
}

def which(name, path):
    for folder in path.split(":"):
        if name in installed.get(folder, []):
            return folder + "/" + name
    return None

print(which("grep", PATH))
print(which("spark-submit", PATH))

What do you expect for grep? And for spark-submit, which is not installed anywhere in this list?

/usr/bin/grep
None

grep is found in the second folder. spark-submit is in none of them, so the answer is None. A real shell would print command not found.

Pipes: the output of one is the input of the next

Here is a small file of orders.

orders.csv, shown as a table. The real file is plain text with commas.

order_idstatusamount
1001paid40
1002refunded15
1003paid22

How many orders are paid? Use two commands joined by a pipe.

ShellCount the lines that mention paid (shown for reading, not run here)
$ grep paid orders.csv | wc -l
2
  1. grep paidkeeps 2 lines
  2. |pipe
  3. wc -lcounts them: 2

grep keeps only the paid lines. The pipe hands them straight to wc -l, which counts lines. No temporary file is made in between.

Each program has three channels: stdin to read from, stdout for results and stderr for errors. A pipe joins stdout to the next stdin. Errors skip the pipe and go to your screen, so a failure upstream can be missed by the steps after it.

Exit codes: how a program says it failed

When a program ends, it hands back a number. By a very old rule, 0 means success, and anything else means something went wrong.

Exit codes you will meet in data work.

CodeUsual meaningWhere you meet it
0Successingest.py finished and wrote its output.
1General failure. grep also uses it when nothing matched.A filter found no records.
2Wrong use, such as a bad option or a missing filegrep was given a file that does not exist.
127Command not foundpython is not on PATH.
137Killed, often for using too much memoryA container ran out of memory.

137 is 128 plus 9, where 9 is the signal that kills a process. In containers it usually means the system stopped your program for using too much memory.

How the code works

A pipeline has several stages, and each returns its own exit code. The whole run worked only if every stage returned 0.

Three stations, one red light
python ingest.py(0)filter (1)gzip (0)

The middle stage exited 1. The chain failed even though gzip returned 0.

Here the filter crashed and exited with 1. gzip then received nothing, compressed nothing, and happily returned 0.

A plain bash pipe reports only the exit code of the last command. So the run looks like a success, although all your records were lost.

Look at the real thing. The file missing.csv does not exist.

ShellThe same pipe, without and with pipefail (shown for reading, not run here)
$ grep paid missing.csv | wc -l
grep: missing.csv: No such file or directory
0
$ echo $?
0
$ set -o pipefail
$ grep paid missing.csv | wc -l
grep: missing.csv: No such file or directory
0
$ echo $?
2

echo $? prints the exit code of the last command. Without pipefail it prints 0, which is wrong, because grep failed. With pipefail it prints 2, the code grep gave.

Production scripts begin with set -euo pipefail. -e stops at the first error, -u treats an unset variable as an error, and -o pipefail makes a pipe fail if any step in it fails.

The rule is easy to write in Python. all() is True only when every item passes.

Pythonpipeline_ok: every code must be 0
def pipeline_ok(exit_codes):
    return all(code == 0 for code in exit_codes)

print(pipeline_ok([0, 0, 0]))
print(pipeline_ok([0, 0, 1]))
True
False

One non-zero code is enough to make the answer False.

Worked examples

Two nightly runs, with the exit code of each stage:

Exit codes of two nightly runs.

StageRun ARun B
extract00
filter01
compress00

Run A is fine. Run B failed, but a plain pass or fail does not say where. A monitor should also name the stage that failed.

PythonName the failed stages
def failed_stages(names, codes):
    failures = []
    for name, code in zip(names, codes):
        if code != 0:
            failures.append({"stage": name, "exit_code": code})
    return failures

names = ["extract", "filter", "compress"]
print(failed_stages(names, [0, 0, 0]))
print(failed_stages(names, [0, 1, 0]))
[]
[{'stage': 'filter', 'exit_code': 1}]

Run A gives an empty list. Run B points straight at the filter stage and its exit code of 1.

Watch outA plain bash pipe hides upstream failures

Without set -o pipefail, bash reports only the last command. Put it at the top of every script that runs without a person watching.

One more problem, to see how a missing program looks. nosuchcmd is not a real command:

ShellCommand not found gives 127 (shown for reading, not run here)
$ nosuchcmd
bash: nosuchcmd: command not found
$ echo $?
127

On a Mac, wc -l prints its number with extra spaces in front. That is only formatting.

Common beginner questions

Why does my script work on my laptop but say command not found on the server?

The program is installed in a folder that is on your laptop's PATH but not on the server's. Install it on the server, or use its full path.

Does a pipe wait for the first command to finish?

No. The commands run at the same time, and data flows through as it is made. That is why a pipe can handle files bigger than memory.

Why does grep return 1 when it finds nothing?

By convention, 1 means no lines matched. It is not a crash, but a script with set -e will still stop on it. That surprises many people.

PATH, pipes and exit codes
  1. Schedulers decide pass or fail from one number: the exit code. 0 is success.

    1. exit code
    2. 0 = success
  2. The shell finds a command by searching the folders in PATH. No match gives 127.

    1. PATH
    2. found or 127
  3. A pipe sends the output of one command into the next.

    1. grep
    2. |
    3. wc -l
  4. A plain pipe reports only the last stage. A failed stage upstream can hide.

    1. stage fails
    2. last stage 0
    3. looks green
  5. set -o pipefail makes the whole pipe fail if any stage fails.

    1. pipefail
    2. exit 2
  6. pipeline_ok is True only when every exit code is 0.

    1. all codes 0
    2. True

What comes next

Next you will run your first Docker container. It packs a program and what it needs into a box, so it runs the same on every machine.

Practice

Run the sample code to see how pipeline_ok treats a list of exit codes.

Write pipeline_ok(exit_codes) to return True only when every number in the list is 0. Call it on [0, 0, 1], save the answer in result, and print it. You should get False.

Going further: chaining commands and reading what failed

Every command ends with an exit code: 0 means success, anything else means failure. && and || use it to decide what runs next, and a pipeline can use it too. Predict which of the echo lines run.

ShellExit codes, &&, || and redirecting errors (illustration, not run here)
false; echo "exit code: $?"
true && echo "ran because the first succeeded"
false && echo "never printed"
false || echo "ran because the first failed"
ls missing_folder 2> errors.txt; echo "ls exit code: $?"
cat errors.txt
printf 'a.csv\nb.csv\n' | xargs -I{} echo processing {}
export STAGE=dev
bash -c 'echo "child sees STAGE=$STAGE"'
exit code: 1
ran because the first succeeded
ran because the first failed
ls exit code: 1
ls: missing_folder: No such file or directory
processing a.csv
processing b.csv
child sees STAGE=dev

false returned a non-zero code, so && skipped its right side and || ran its right side. The failed ls wrote its message to errors.txt because of 2>, and its exit code (1 on this system, 2 on some others) is still available. xargs turned each input line into one command.

And export made STAGE visible to the child process, which a plain assignment would not.

PatternMeaningUse
a && bRun b only if a succeededSteps that depend on each other
a || bRun b only if a failedA fallback or an alert
a; bRun b either wayIndependent steps
export NAME=valuePass a variable to child programsConfiguration for scripts

The exact wording of error messages and the number a failing command returns vary between systems, but the rule is stable: zero is success.

In a script, set -e stops at the first failure, and set -o pipefail makes a pipeline fail if any stage fails, so a broken first stage cannot hide behind a successful last one.

Try a new scenario

A nightly script runs: download_data; transform_data; publish_report. The download fails but the report is still published. What is wrong and how do you fix it?

Work it out, then open the answer

The semicolons run every step regardless of earlier failures, so transform and publish used stale or missing data. Chain the steps with && so each one runs only if the previous one succeeded, or start the script with set -e so it stops at the first failure. Then a failed download leaves yesterday's report in place instead of publishing a wrong one.

What does set -e do in a script?

It makes the script stop at the first command that fails, so later steps do not run on broken input. Pair it with pipefail so a failure inside a pipeline is not hidden.

Close the lesson and explain it

Without scrolling up, explain the difference between && and ;, and what a non-zero exit code means.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?