Day 30Phase 06 · Airflow + Orchestration

Production Airflow

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Explain logical dates (execution date) and data intervals to orchestrate deterministic historical backfills.
  • Configure task retries with exponential backoff and execution timeouts to prevent hung worker slots.
  • Distinguish sensor poke mode from reschedule mode to avoid worker slot starvation.

Task Isolation and Process Boundaries

In a single Python script, functions share the same memory space. If Function A modifies a global variable, Function B can read that modified variable immediately.

In Apache Airflow, this is never true. Airflow enforces strict task isolation:

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: Logical date vs Execution date vs Data interval, catchup=False vs catchup=True, CLI backfills & idempotent rerun safety, Sensor poke vs reschedule mode, Worker slot starvation prevention, Retries with exponential backoff & jitter, Avoiding top-level Variable.get() / Connection.get() DB blasts

Build: Production Airflow simulator: logical date intervals, sensor slot starvation remediation, exponential retry jitter, and execution_timeout circuit breakers.

Never use sensor poke mode for long waits (>5 min): it locks worker slots and starves all other pipelines. Always use mode='reschedule'.

Already have a pass? Go to your account. Need a pass? See pricing.