Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

Spark architecture (Driver, Cluster Manager, Executors)

Easyarchitecturedriverexecutorcluster-manager

Interview question

Explain Spark architecture: Driver, Cluster Manager, and Executors.

How to answer

The short answer

Spark jobs run as a coordinated distributed system with three main roles.

Components

  1. 1Driver: runs your main program, builds the logical/physical plan, schedules tasks, tracks progress, and collects small results.
  2. 2: grants CPU/memory (YARN, Kubernetes, or Spark Standalone).
  3. 3Executors: JVM processes on worker nodes that run tasks, store data, and hold cached .

What to say in an interview

  • The driver is the brain; executors are the workers.
  • If the driver dies, the application usually fails.
  • Executor failure is recoverable for recomputable RDDs/ (lineage).
  • Keep the driver light: do not collect() huge datasets to the driver.

Practical implication

Python
# Bad: pulls all rows to the driver
rows = df.collect()

# Better: aggregate on executors, bring back a small result
summary = df.groupBy("country").count().collect()

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill