The short answer
Spark jobs run as a coordinated distributed system with three main roles.
Components
+------------------+
| Cluster Manager |
| (YARN/K8s/Stand) |
+--------+---------+
| allocates
v
+-------------+ +------+------+ +-------------+
| Driver |---->| Executor 1 | | Executor 2 |
| (your app) | | tasks+cache | ... | tasks+cache |
+-------------+ +-------------+ +-------------+- 1Driver: runs your main program, builds the logical/physical plan, schedules tasks, tracks progress, and collects small results.
- 2: grants CPU/memory (YARN, Kubernetes, or Spark Standalone).
- 3Executors: JVM processes on worker nodes that run tasks, store data, and hold cached .
What to say in an interview
- The driver is the brain; executors are the workers.
- If the driver dies, the application usually fails.
- Executor failure is recoverable for recomputable RDDs/ (lineage).
- Keep the driver light: do not
collect()huge datasets to the driver.
Practical implication
# Bad: pulls all rows to the driver
rows = df.collect()
# Better: aggregate on executors, bring back a small result
summary = df.groupBy("country").count().collect()