Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Secrets backends

Airflow & DAGs · Operating Airflow in Production

Secrets backends

Mediumairflow-59
secrets-backendsecurityvaultaws-secrets-manager

Question

How do you plug Airflow into a secret manager?

Solution

A secrets backend connects Airflow directly to enterprise secret managers, allowing credentials and configuration variables to be managed securely without storing plaintext secrets in the metadata database.

Why direct database storage is risky

In standard Airflow setups, connections and variables are saved in the internal PostgreSQL or MySQL metadata database. Even though passwords can be encrypted using a Fernet key, this creates operational risks:

  • Database backups and snapshot dumps contain sensitive database credentials and API keys.
  • Rotating an AWS key or database password requires updating entries manually through the Airflow web UI.
  • Managing multiple development, staging, and production environments leads to credential drift.

How secrets backends operate

Airflow can query external secret managers like AWS Secrets Manager, Google Cloud Secret Manager, or HashiCorp Vault. Configuration is set in airflow.cfg or via environment variables:

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}

When an operator requests a connection named snowflake_analytics, Airflow follows a strict lookup hierarchy:

Lookup resolution order:                                                     1. Secrets Backend (AWS Secrets Manager / Vault / GCP) ──> Found? Use it     2. Environment Variables (AIRFLOW_CONN_SNOWFLAKE_ANALYTICS) ──> Found? Use it3. Metadata Database (Postgres connection table) ──> Fallback                

Benefits and API caching

Using an external backend allows infrastructure teams to enforce enterprise security policies: access logs are audited, secrets can be rotated automatically without code changes, and sensitive tokens remain hidden from web UI viewers.

To prevent high volumes of task runs from overwhelming cloud providers with API queries and incurring steep API charges, configure the backend's caching parameters (such as backend_kwargs={"caching": True}) to cache credentials locally for worker processes.

PreviousNext