Modern stack
Apache Airflow for Data Engineers
DAGs, operators, scheduling, retries and backfills in Apache Airflow.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Engineers who write or maintain Airflow pipelines and candidates who get orchestration questions in interviews.
What you will be able to do
- Write a DAG with the TaskFlow API and pass data between tasks
- Understand scheduling, data intervals and backfills
- Pick an executor and scale tasks without overloading the scheduler
- Design idempotent pipelines that are safe to retry
What's inside
Pick a chapter and the notebook opens right there.
- Architecture & Airflow 3.xScheduler, DAG Processor, Triggerer, API Server & Metadata DB; decoupled Task SDK, Execution API, worker isolation, heartbeat misses, zombie task triage & HA multi-scheduler designStart here
- DAG Authoring & TaskFlow APITaskFlow API (@task) vs classic operators, type annotations, TaskGroups UI nesting, default_args inheritance, DAG parameters (Params/UI), versioning & Git-sync multi-team bundlesStart here
- Scheduling Semantics & TimetablesLogical date vs Data Interval (interval_start, interval_end), cron vs custom Timetables, asset-based reactive triggers, catchup storms, max_active_runs & DST timezone trapsStart here
- Task State & Control FlowTrigger rules (all_success, all_done, none_failed_min_one_success), branching (@task.branch), ShortCircuitOperator, depends_on_past deadlocks, exponential retries & SLA alertsStart here
- Data Passing & Dynamic Task MappingXCom fundamentals & DB limits, custom cloud XCom backends (S3/GCS), dynamic task mapping (expand, partial, zip), map-reduce patterns & scaling to 10k instances safelyStart here
- Sensors & Deferrable Operatorsmode='poke' slot starvation vs mode='reschedule', async Deferrable Operators with Triggerer asyncio daemon, ExternalTaskSensor vs ExternalTaskMarker cross-DAG triggersStart here
- Executors & Horizontal ScalingLocal vs Celery vs KubernetesExecutor vs Edge; K8s ephemeral pods & startup latency, concurrency knobs (pools, priority_weight, max_active_tasks) & scheduler loop tuningStart here
- Operators & DE Ecosystem ProvidersSQLExecuteQuery, Snowflake, BigQuery, Databricks, KubernetesPodOperator container isolation, custom operators/hooks, circuit breakers & dbt/Spark orchestration patternsStart here
- Secrets, Connections & PerformanceUI vs ENV vs Secrets Backends (AWS, Vault, GCP); prefix resolution, IAM roles, parsing performance benchmarks,.airflowignore & slashing parse latency from 30s to 200msStart here
- Idempotent Pipeline PatternsMathematical idempotency, partition overwrite (INSERT OVERWRITE), transactional staging swaps, backfill-safe engineering, late-arriving data & zero-downtime view pointersStart here
- Testing & Local Development CI/CDTesting pyramid, DAG integrity tests (no cycles, valid structure), unit testing tasks with dag.test(), mocking hooks/fixtures, Astro CLI dev rigs & GitHub Actions CI gatesStart here
- Deployment & Managed AirflowExtending official Docker images, production Helm chart tuning (PDBs, HPA), managed services matrix (MWAA vs Cloud Composer vs Astro), zero- downtime DB migrations & blue-green DAG deploysStart here
- Observability & OperationsRemote logging (S3, GCS, CloudWatch), StatsD/OpenTelemetry metrics, Grafana alerting, zombie pod triage, metadata DB maintenance (airflow db clean, VACUUM) & OpenLineage/MarquezStart here
- Security & Multi-TenancyRBAC role hierarchies, DAG-level access controls, modern Auth Managers (Keycloak, Okta, IAM Identity Center), hard K8s multi-tenant namespaces, network isolation & SOC2 audit loggingStart here
- Airflow vs Modern OrchestratorsComparative evaluation: Airflow vs Dagster (software- defined assets) vs Prefect vs Step Functions vs Databricks Workflows; migration runbooks & when NOT to use AirflowStart here
- Staff Drills & Cheat Sheet500-DAG platform architecture, 2-year backfill incident postmortem, slashing managed Airflow spend by 60%, Tier-0 SLA design, Senior DE interview deep-dive & Airflow 3.x Staff Cheat SheetStart here
Practice this
Put it to work on LakeBench.
Read the guide first
Short explainers that pair with this notebook.
Common questions
- Does it cover Airflow 3?
- Yes. The first chapter covers the Airflow 3.x architecture, and the later chapters use current patterns.
- How does it compare Airflow with other orchestrators?
- A chapter compares Airflow with modern orchestrators, so you can answer the common interview question about when to use which.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
