Dynamic DAGs from config

Dynamic DAG generation creates dozens or hundreds of similar pipelines programmatically from configuration files such as YAML, JSON, or database tables, rather than hand-writing separate Python DAG files for each table.

Generation patterns

There are two common implementation patterns:

• Single-file loop: A single Python script reads local configuration files, iterates over the definitions, creates DAG objects, and registers them in Python's global namespace:

# Iterating over configs to register DAGs
for config in table_configs:
    dag_id = f"sync_{config['table_name']}"
    dag = DAG(dag_id=dag_id, schedule=config["schedule"], default_args=default_args)
    # attach operators
    globals()[dag_id] = dag

Other teams generate concrete files ahead of deployment:

• Code generation at build time: A CI/CD step or pre-commit hook runs a Jinja template to generate concrete .py files on disk before deploying them to the repository.

What can go wrong in production

While dynamic generation reduces boilerplate, it introduces subtle operational hazards:

The primary risk is parse-time degradation. If a Python script generates 150 DAGs, the Airflow DAG processor executes that script on every parsing cycle. If the script queries an external metadata catalog or calls a REST API to find table names, it introduces network latency into the scheduler loop and risks scheduler parse timeouts. Config lookups must read from local files, not remote networks.

Colliding identifiers represent another pitfall. Every generated DAG must guarantee a strictly unique dag_id, or Airflow will overwrite definitions silently.

Blast radius is also amplified. A syntax error, malformed YAML file, or unhandled null field in the shared factory script breaks parsing for all generated DAGs at once, preventing any of them from scheduling.

In Airflow 3, DAG bundles help mitigate this by versioning and isolating configuration repositories independently from core orchestrator code.