When the Airflow scheduler is slow to transition tasks from scheduled to running, the bottleneck usually stems from top-level file parsing overhead, database connection contention, or sub-optimal scheduler loop settings.
1. Optimize DAG file parsing
Slow parsing is the most frequent cause of scheduler lag. The scheduler must parse every DAG file repeatedly:
- Profile parsing performance using the CLI command
airflow dags reportto identify files taking longer than 500 milliseconds to parse. - Eliminate top-level database queries, REST API calls, and heavy imports.
- Place an
.airflowignorefile in folders containing helper scripts, test suites, or documentation so the DAG processor does not waste cycles parsing non-DAG Python files.
2. Tune scheduler processes and intervals
Adjust concurrency settings in airflow.cfg:
parsing_processes: Increase from the default of 2 to match the available CPU cores on the scheduler host (for example 4 or 8), allowing parallel file parsing.min_file_process_interval: Increase from 30 seconds to 60 or 90 seconds in stable production environments. This prevents the scheduler from consuming 100% CPU constantly re-parsing DAGs whose code changes infrequently.
3. Deploy multi-scheduler high availability
Since Airflow 2.0, teams can run multiple active scheduler instances in parallel against the same metadata database. Running two or three scheduler replicas distributes task scheduling, eliminates single points of failure, and reduces task dispatch latency during high-volume spikes.
4. Maintain metadata database health
A bloated metadata database severely degrades scheduler performance. The scheduler continuously executes SELECT and UPDATE queries against the task_instance and dag_run tables.
Implement periodic maintenance by running airflow db clean to purge task instances older than 60 or 90 days. Ensure the database has appropriate connection pooling through PgBouncer and sufficient IOPS to prevent slow database transactions from blocking the scheduler main loop.