Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Cluster managers

PySpark · Execution Model

Cluster managers

Easypyspark-44
cluster-manageryarnkubernetesstandalone

Question

Which cluster managers can run Spark, and how do they differ?

Solution

Spark does not manage machines itself. It asks a cluster manager for containers, and the manager decides where they run. The common choices are Standalone, YARN and Kubernetes. Mesos was an option before but is deprecated, so you would not start a new project on it.

What each one is

  • Standalone: Spark's own simple manager. Easy to start on a few machines. Few features for sharing a cluster between teams.
  • YARN: the resource manager from the Hadoop world. It dominates in on-premises Hadoop shops and in older EMR and Dataproc setups. It has queues and capacity rules for many teams.
  • Kubernetes: Spark driver and executors run as pods. A good match if the company already runs everything in containers, and it gives clean dependency isolation, because each job brings its own image.

Managed services

Most teams do not set up any of these by hand. Databricks, Amazon EMR, Google Dataproc and Azure Synapse or HDInsight offer Spark as a service. They pick the cluster manager for you and add autoscaling, monitoring and job scheduling. In an interview, "we ran Spark on Databricks" or "on EMR with YARN" is a perfectly good answer.

What actually differs for you

Your job -> asks for N executors of X cores and Y GB
Cluster manager -> finds space, starts containers
Spark -> schedules tasks on those containers

Spark's code and APIs are the same on all of them. What changes is how you submit (--master yarn vs k8s://...), how dependencies are shipped (zip files vs container images), how logs are fetched, and how resources are shared with other workloads.

A short answer for the interview: Spark only requests resources, the manager hands them out, and YARN and Kubernetes are the two you will meet most.

🎯 Put this concept into practice

Solidify this answer with real hands-on interview drills in the browser studio.

Open related drill →
PreviousNext