The short answer
Running Apache Spark workloads on Google Cloud has traditionally involved creating provisioned Google Cloud Dataproc clusters composed of primary master nodes and secondary worker Compute Engine VMs.
While effective, provisioned clusters introduce operational overhead:
- Clusters take 3 to 5 minutes to provision and bootstrap.
- If clusters run 24/7 to handle periodic scheduled jobs, organizations pay continuous VM infrastructure fees for idle compute.
- Sizing clusters requires guessing peak worker counts.
Architecture of Dataproc Serverless
Dataproc Serverless runs Spark batch workloads without requiring engineers to provision, manage, or tune Compute Engine clusters.
When submitting a batch job (such as a PySpark transformation script):
- You submit the code directly via Google Cloud CLI, Cloud Composer, or .
- Dataproc Serverless provisions a dynamically managed Spark runtime on Google's internal infrastructure within seconds.
- The runtime automatically provisions and scales executors dynamically based on stage volume.
- When the batch script completes, all compute resources terminate immediately.
# Submit PySpark batch directly without managing any cluster
gcloud dataproc batches submit pyspark gs://my-lake/scripts/transform_sales.py --project=my-data-project --region=us-central1 --deps-bucket=gs://my-lake/dependencies --version=2.2Dependency management with custom containers
In traditional clusters, installing specialized Python libraries (such as proprietary wheel files or machine learning packages) required writing complex initialization actions that executed on every VM startup.
Dataproc Serverless supports custom container images. Data engineers build standard containers packaged with required libraries, push them to Google Artifact Registry, and pass --container-image during job submission. Jobs execute in a pre-compiled environment with zero installation delay.