On this page
Imagine a nightly job with 3 steps: collect yesterday's orders, clean them, and update the sales report. You could run each step by hand. You could also set an alarm and a reminder. But what happens if step one fails at 2 am? Does step two still run on empty data? Who finds out? Apache Airflow exists to answer those questions. It is a tool that runs your steps on a schedule, in the right order, and deals with failure.
Airflow is open source, and it appears in a large share of data engineering job descriptions, which is why beginners keep hearing the name. The example code in this guide is illustrative. It was not run here, because running Airflow needs its own setup, so check the details against the documentation for your version.
Read yesterday's orders.
retries=2Drop rows with no amount.
Write to the warehouse.
It runs every day. If extract still fails after its retries, transform and load do not start.
The vocabulary
- DAG. Short for Directed Acyclic Graph, which sounds worse than it is. It is the plan for a pipeline: a set of tasks, with arrows showing which must finish before which. "Directed" means the arrows have a direction. "Acyclic" means no loops, so a task can never end up waiting on itself.
- Task. One step: run a SQL query, run a Python function, copy a file.
- Schedule. When the DAG runs: every day, every hour, or only when you trigger it.
- Run. One execution of the DAG for one time period. Airflow keeps a record of every run and every task inside it.
- Retry. If a task fails, Airflow can try it again after a pause, a set number of times, before giving up and alerting you.
A small example
DAGs are written in Python. Here is one in the style of recent Airflow 3 releases, using the @dag and @task decorators. Each @task function is a step, and passing one task's result to the next sets the order.
from datetime import datetime
from airflow.sdk import dag, task
@dag(schedule="@daily", start_date=datetime(2026, 3, 1), catchup=False)
def daily_orders():
@task(retries=2)
def extract():
# in real life: read yesterday's orders from the source
return [{"order_id": 1, "amount": 1200}, {"order_id": 2, "amount": 800}]
@task
def transform(rows):
return [row for row in rows if row["amount"] > 0]
@task
def load(rows):
print(f"loaded {len(rows)} rows")
load(transform(extract()))
daily_orders()Read it top to bottom. @dag(schedule="@daily", ...) says this runs once a day. extract, transform and load are the 3 tasks, and load(transform(extract())) says the order: extract first, then transform, then load. retries=2 means that if extract fails, Airflow tries it up to two more times. In older Airflow 2 versions, the imports and the way you write DAGs differ a little, so check which version your team uses.
What runs the DAG
Airflow is a small system of several parts working together. Knowing them helps you read the web page and the errors.
Reads your DAG files.
Hands tasks to workers.
Execute your Python, SQL or commands.
Every run, task and status.
Runs, logs, and a rerun button.
- The scheduler reads your DAG files, decides which runs and tasks are due, and hands them out.
- Workers (run by an executor) actually run the tasks.
- The metadata database remembers every run, task and status.
- The web interface shows your DAGs and runs, lets you rerun a failed task, and shows logs.
What Airflow does not do
This is the most useful thing a beginner can learn. Airflow is an orchestrator: a conductor, not a musician. It does not transform your data by itself. It tells other things to: a Python function, a SQL query in the warehouse, a Spark job, a dbt command. Putting heavy data processing inside the Airflow task itself is a common beginner mistake. The task should start the work somewhere that is built for it, and wait.
Retries and reruns: why idempotency matters
Because Airflow retries failed tasks and can rerun past days (a backfill), every task will sometimes run twice for the same day. If running twice doubles your data, the pipeline is broken. The fix is to make each task idempotent, meaning the result is the same however many times it runs, as shown with a runnable example in what is a data pipeline. The catchup=False setting in the example stops Airflow from automatically running every missed day since the start date, which is a common surprise.
Do you always need Airflow?
No. A single script that runs once a day can be scheduled with a plain cron job, the simple built-in scheduler on Linux and Mac computers. Airflow earns its place when you have many steps with dependencies, need retries and alerts, want a history of runs, and need to rerun a past day easily. Alternatives include Dagster, Prefect, and the managed pipeline tools of the big clouds. The ideas in this guide, tasks, dependencies, schedules, retries and idempotency, apply to all of them.
What to learn next
- What is a data pipeline, to see what a task actually does.
- Python for data engineering beginners, since DAGs are Python.
- The Orchestration & Reliable Pipelines track for DAGs, retries, backfills and idempotency in practice, and the Airflow interview questions once you are ready for interviews.