Deferrable operators suspend their execution, yield their worker slot, and hand off monitoring to an asynchronous service called the Airflow Triggerer, freeing up worker capacity while waiting for external jobs.
The problem with classic worker waiting
When an operator waits for an external system (such as an EMR Spark job, a Databricks run, or an S3 file), a standard worker task remains in the running state. The worker thread sits idle in a time.sleep() loop.
In large organizations where hundreds of pipelines wait for warehouse jobs, hundreds of worker slots remain locked doing zero CPU work. This forces teams to over-provision expensive worker clusters just to hold idle waiting threads.
How deferrable operators work
Deferrable operators use Python's asyncio event loop:
Worker process:
1. Task starts on worker, submits job to external API
2. Task raises TaskDeferred event, yielding its worker slot completely
↓
Triggerer process:
3. Triggerer (single asyncio daemon) monitors job status asynchronously
4. When job completes, Triggerer fires an event back to the metadata DB
↓
Scheduler:
5. Scheduler re-queues task to resume on any available workerA single Triggerer process running on modest hardware can monitor thousands of concurrent deferred tasks simultaneously using asynchronous HTTP calls.
Reschedule mode versus deferrable
Sensors configured with mode="reschedule" release worker slots between checks, but they create constant scheduling overhead. Every five minutes, the sensor must be re-scheduled, assigned to a worker, boot up a Python process, run one check, and shut down.
Deferrable operators avoid this churn. The task leaves the worker completely and resumes only once, upon final completion. Most modern provider packages (such as Snowflake, Databricks, and BigQuery) now offer deferrable=True flags out of the box.