Day 31Phase 06 · Airflow + Orchestration
Airflow + Spark + Cloud
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Enforce separation of concerns: Airflow orchestrates workflows while external engines (Spark, EMR, BigQuery) compute.
- Dispatch asynchronous jobs to external compute clusters without locking Airflow worker processes.
- Implement atomic staging partition swaps to ensure consumers never observe partial write states.
Conditional Branching with @task.branch
Decorating a Python function with @task.branch enables runtime DAG route selection.
The branch function returns the exact string ID of the task that Airflow should schedule next.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Separation of concerns: Airflow orchestrates, Spark computes, Why running PySpark code inside Airflow workers crashes worker pods, SparkSubmitOperator vs EMR / Dataproc / Synapse / Databricks operators, Deferrable operators & asynchronous triggers, S3/ADLS/GCS staging handoff: Extract → Raw → Spark → Curated → Warehouse COPY/MERGE, Atomic partition replacement & staging cleanup
Build: Cloud lakehouse orchestrator (AWS, Azure, GCP): Airflow async trigger DAG → EMR/Dataproc/Synapse Spark execution → object storage Parquet validation → atomic warehouse MERGE gate.
Never execute heavy compute on Airflow worker nodes. Treat Airflow strictly as the air traffic controller dispatching jobs to managed clusters.
Already have a pass? Go to your account. Need a pass? See pricing.