Any Python code written outside of task callables runs on every single DAG parsing cycle, which happens every few tens of seconds. Placing database queries, API calls, or heavy computation at the top level degrades scheduler performance and can take down production systems.
The DAG parsing cycle
The Airflow DAG processor runs an endless loop scanning the DAGs directory. Controlled by min_file_process_interval (which defaults to 30 seconds), it executes every Python file to construct the in-memory task dependency graph:
If a DAG file makes a database connection or calls requests.get() at the top level, that network call executes every 30 seconds for that file alone. With 100 DAG files, the scheduler generates thousands of unnecessary database connections per hour, exhausting connection pools and causing CPU spikes.
Import timeouts
If an external API experiences latency and takes 40 seconds to respond, the DAG processor process exceeds dagbag_import_timeout (typically 30 seconds). The scheduler marks the file with a parse timeout error and drops the DAG from the UI, preventing scheduled tasks from queuing.
How to write clean DAG files
Follow these practices to keep parsing under 100 milliseconds per file:
- Keep top-level code strictly declarative, creating only DAG and Operator instances.
- Move all business logic, database queries, and third-party imports inside task functions or operator
execute()methods. - Avoid calling
Variable.get()at the top level. Instead, use Jinja template references like{{ var.value.my_variable }}, which Airflow evaluates only when the task actually executes on a worker.