Packaging and dependency management

Pin every dependency to an exact version in a lock file, declare them in pyproject.toml, keep development tools separate from runtime ones, and run the job in the same container image everywhere. That way the code behaves the same on your laptop, in CI, and on the production worker.

Why it breaks otherwise

If you write pandas in a requirements file with no version, then two machines installing at different times get different versions. A new release changes a default, a deprecated function disappears, or a library you depend on updates its own dependencies, and a job that worked yesterday fails today. "It works on my machine" is the symptom. On Airflow, this shows up when the scheduler, the workers and your laptop have different libraries.

Declare and lock

• pyproject.toml lists the direct dependencies with reasonable ranges (pandas>=2.1,<3).
• A lock file records the exact versions of everything, including indirect dependencies, with hashes. Tools that produce one: uv (uv.lock), Poetry (poetry.lock), pip-tools (pip-compile creates a pinned requirements.txt).
• Install from the lock file in CI and production, so the result is exactly the tested set.

Separate development dependencies

Testing, linting and notebook tools (pytest, ruff, jupyter) go in a dev group. The production image installs only the runtime set. That makes it smaller, quicker to start, and less exposed to vulnerabilities.

Use the same runtime everywhere

A Docker image bundles the Python version, system libraries (like libpq or libgomp) and the packages. Build it in CI, tag it with the commit, and run that exact image on workers, whether in Airflow's KubernetesExecutor, Cloud Run, or ECS. Pin the base image version too, as the system libraries and the Python version matter.

Good habits

• Use a virtual environment for each project (uv venv, python -m venv), never the system Python.
• Update dependencies on purpose: a scheduled bot (Dependabot or Renovate) opens pull requests, CI runs the tests, and you merge the ones that pass.
• Pin the Python version (requires-python, or .python-version).
• Run a vulnerability scan (such as pip-audit) in CI.

How to answer

Say "lock file plus container". Name one tool you have used, and say what failure it prevents, in terms of a job that suddenly broke after a library update.