Config and secrets management

Keep code and configuration separate, load them from the environment, keep secrets in a secret manager, and validate the config at startup. The same code should run in dev, staging and production with only the configuration changing.

Separate config by environment

Things that differ between environments, such as the database host, bucket names, or the schema to write to, belong in configuration, not in if env == "prod" branches scattered through the code. A common setup is a base config plus an override for each environment, or just environment variables set by the deployment.

Environment variables and .env files

import os
db_host = os.environ["DB_HOST"]          # fails clearly if missing
batch_size = int(os.getenv("BATCH_SIZE", "1000"))

For local development, a .env file (loaded with python-dotenv) is convenient. Add it to .gitignore. In production, the platform (Kubernetes, Airflow, a CI system) injects the variables, and no .env file should exist.

Secrets

Passwords, API keys and tokens should live in a secret manager (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault, Azure Key Vault), and the job fetches them at runtime using its own identity (an IAM role or service account). Benefits: access is controlled and audited, and secrets can be rotated without code changes. Environment variables are fine as a delivery mechanism, as long as the value comes from the manager.

Validate on startup

A typo in a config value should stop the job in the first second, not after two hours. pydantic-settings lets you declare the settings with types, defaults and required fields:

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    db_host: str
    db_port: int = 5432
    batch_size: int = 1000

settings = Settings()      # reads environment variables, raises if invalid

Do not

Commit keys to Git (they stay in history even after you delete the file), print secrets in logs, or pass them on command lines, which show up in process listings. If a secret leaks, rotate it at once; deleting the commit is not enough.

Mention that a secret scanner in CI (such as gitleaks) catches mistakes before they reach the repository.