The deploy mode decides where the driver process runs. In client mode, the driver runs on the machine that submitted the job. In cluster mode, the cluster manager starts the driver inside the cluster, on one of its nodes. Executors always run in the cluster either way.
What changes
client mode: [your laptop / edge node: DRIVER] <--network--> [executors in cluster] cluster mode: [laptop only submits] [cluster node: DRIVER] <--> [executors]
In client mode, the driver's console output comes straight to your terminal, and you can use it from a notebook or pyspark shell. That is why interactive work uses client mode. The price is that the job is tied to that machine. If the laptop sleeps, loses VPN, or the SSH session drops, the driver dies and the whole application with it.
In cluster mode, you submit and walk away. The driver lives next to the executors, so a closed laptop does not matter, and driver-to-executor traffic stays on the cluster network. Logs are then on the cluster, so you fetch them from the resource manager or log store.
Network cost
The driver talks to executors constantly: task descriptions, results, broadcast data, collect() output. If the driver is far away from the cluster, say a laptop on home wifi, every collect and every broadcast crosses a slow link. Jobs that feel fine in a notebook with small data can crawl.
Which to use
- Notebooks, debugging, short experiments: client mode.
- Scheduled production jobs from Airflow or similar: cluster mode.
One detail worth knowing: Spark standalone mode does not support cluster mode for Python applications. YARN and Kubernetes do. Managed services such as Databricks and EMR make this choice for you in most cases.