Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Partitioning & Pruning

What is an executor?

Mediumexecutorarchitecturecluster

Interview question

What is an executor in Spark?

How to answer

The short answer

An executor is a JVM process launched on a worker node for the lifetime of a Spark application. It runs tasks, stores data, and can hold cached .

Responsibilities

Output
Executor
  |- task threads (cores)
  |- memory regions (execution + storage)
  |- disk for spill / shuffle / cache overflow
  |- reports metrics/status to the driver

How it relates to resources

  • cores: parallel task slots in that executor
  • memory: heap (and sometimes off-heap) for execution and cache
  • Multiple executors per node are common depending on cluster config
Python
# Common submit-time settings (cluster dependent)
# --executor-memory 8g
# --executor-cores 4
# --num-executors 50

Failure behavior

If an executor dies, Spark reschedules its tasks on other executors using lineage (for recomputable data). Cached blocks on the dead executor are lost and may be recomputed.

Interview tip

Contrast driver (coordinates) vs executor (computes). Mention that Python also involve Python workers alongside the JVM executor.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill