Day 30Phase 06 · Airflow + Orchestration
Production Airflow
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Explain logical dates (execution date) and data intervals to orchestrate deterministic historical backfills.
- Configure task retries with exponential backoff and execution timeouts to prevent hung worker slots.
- Distinguish sensor poke mode from reschedule mode to avoid worker slot starvation.
Task Isolation and Process Boundaries
In a single Python script, functions share the same memory space. If Function A modifies a global variable, Function B can read that modified variable immediately.
In Apache Airflow, this is never true. Airflow enforces strict task isolation:
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Logical date vs Execution date vs Data interval, catchup=False vs catchup=True, CLI backfills & idempotent rerun safety, Sensor poke vs reschedule mode, Worker slot starvation prevention, Retries with exponential backoff & jitter, Avoiding top-level Variable.get() / Connection.get() DB blasts
Build: Production Airflow simulator: logical date intervals, sensor slot starvation remediation, exponential retry jitter, and execution_timeout circuit breakers.
Never use sensor poke mode for long waits (>5 min): it locks worker slots and starves all other pipelines. Always use mode='reschedule'.
Already have a pass? Go to your account. Need a pass? See pricing.