Airflow & DAGs · Architecture & Scalability

How does Airflow handle logs?

Mediumloggings3ops

Interview question

How does Airflow handle logs in distributed deployments?

How to answer

The short answer

With many workers (or short-lived K8s pods), logs shouldn't live only on one machine.

Remote logging sends logs to centralized storage (S3, GCS, Azure Blob) so they remain available after workers disappear.

Interview tip: Remote logging is especially important with KubernetesExecutor.