Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Cloud Composer specifics

Airflow & DAGs · Operating Airflow in Production

Cloud Composer specifics

Mediumairflow-70
cloud-composergcpmanaged-servicesarchitecture

Question

What is different about running Airflow on Cloud Composer?

Solution

Cloud Composer is Google Cloud's fully managed workflow orchestration service built on Apache Airflow. It runs Airflow on Google Kubernetes Engine (GKE) and integrates natively with Google Cloud services like BigQuery, Cloud Storage, and IAM.

Key architectural differences from vanilla Airflow

Running on Composer introduces several operational distinctions:

  • GCS bucket synchronization: DAG files and custom plugins are not deployed directly to server filesystems. Instead, developers upload files to a designated Google Cloud Storage bucket (gs://composer-bucket/dags). A background sync daemon continuously copies files from the bucket to GKE worker pods.
  • Composer 3 generation: Cloud Composer 3 is the current platform generation. It uses a decoupled control plane, provides serverless autoscaling workers, and supports recent Airflow versions. Which exact Airflow versions an environment can run changes with each Composer release, so check Google's version list before you plan an upgrade, especially a move to Airflow 3.
  • Native IAM integration: Tasks authenticate to Google Cloud APIs (such as BigQuery, Dataproc, and Vertex AI) using the environment's attached Google Cloud service account, eliminating the need to store credentials in Airflow Connections.

Managing Python dependencies

Installing custom PyPI packages on Composer requires updating the environment configuration via the Google Cloud Console or the gcloud CLI:

gcloud composer environments update my-env \
    --location us-central1 \
    --update-pypi-packages-from-file requirements.txt

This triggers an automated Cloud Build process that creates a new container image. If third-party packages introduce conflicting transitive dependencies with pre-installed Google Cloud provider libraries, the build fails and the environment rolls back.

Cost considerations

Cloud Composer provides high availability and managed operations, but it maintains persistent control plane infrastructure (including Cloud SQL, GKE control planes, and API components). Because of these baseline components, Composer incurs ongoing hourly infrastructure costs even when no tasks are running, unlike fully serverless event-driven tools.

PreviousNext