The short answer
Google Cloud Composer is a managed workflow orchestration service built on open-source Apache Airflow. While Composer 1 deployed Airflow components onto standard customer-managed Google Kubernetes Engine (GKE) worker nodes, Composer 2 overhauled this architecture using GKE Autopilot.
The Composer 2 infrastructure stack
A Cloud Composer 2 environment separates orchestration into isolated components:
- Tenant Project: Google provisions a fully managed tenant project containing the Cloud SQL metadata database and Airflow web server.
- Customer Project: Customer GKE Autopilot cluster hosts the Airflow schedulers, Celery triggerer pods, and Airflow worker containers.
- Cloud Storage Bucket: Environment bucket mounted to all pods via Cloud Storage FUSE, ensuring DAG file changes sync across schedulers within seconds.
Dynamic worker autoscaling with CeleryExecutor
In legacy Airflow deployments, workers were static VMs: if 50 tasks ran simultaneously during morning batch windows, jobs queued; during the afternoon, idle worker VMs continued billing.
Composer 2 integrates the Airflow CeleryExecutor with Kubernetes Horizontal Pod Autoscaling (HPA):
- You define worker limits (for example, minimum 2 workers, maximum 20 workers).
- When DAG runs schedule dozens of concurrent tasks, the Airflow scheduler monitors queue depth.
- GKE Autopilot provisions additional worker pods in seconds to absorb task volume.
- As tasks finish and the queue clears, worker pods scale down to the configured minimum, eliminating idle compute expenses.
Best practice: Light workers, heavy offloading
An architectural rule in Cloud Composer is keeping Airflow workers lightweight. Airflow should act purely as an orchestrator, delegating heavy computational processing to serverless compute engines (BigQuery, Dataproc Serverless, or Cloud Dataflow) via operator hooks rather than processing gigabytes of data inside PythonOperator memory.