Driver, executors & the cluster
Know who plans the work and who actually runs it.
The driver coordinates your application. Executors run tasks and hold application data. The cluster manager allocates resources.
Three roles to keep separate
The driver plans and schedules work. The executors are application processes that run tasks. A cluster manager, such as YARN, Kubernetes, or Spark standalone, provides resources. A worker machine is not the same thing as an executor process.
Where does your Python run?
In classic PySpark, your Python program communicates with Spark’s JVM machinery. Python UDFs can involve Python worker processes on executors. Built-in DataFrame expressions generally execute through Spark’s engine; writing Python syntax does not mean every row runs through Python.
Think about the destination
A distributed operation can finish successfully and still overwhelm the driver if you bring back too much data. When debugging, inspect a bounded sample and the Spark UI. Ask which process ran out of memory before suggesting more executor memory.
Try it on a small dataset
Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.master("local[2]") # two local task threads, not two executors
.appName("spark-fieldnotes")
.getOrCreate())
spark.range(10).show(5)Shows IDs 0–4, followed by a notice that only five rows are shown.
Result reasoned through on the small input; not runtime-tested against a Spark cluster.
local[2] is a local learning setup, not a two-machine cluster. In production, deployment configuration determines where the driver runs.
The driver crashes after collect(), but executors have free memory. Why?
Say your answer out loud before opening the notes.
Reveal the reasoning
collect() returns all rows to the driver. Diagnose the driver result size and heap rather than only executor capacity. Keep processing distributed, write results to storage, or inspect a bounded sample.
Sources & verification
Core claims checked against Apache Spark 3.5.7 documentation and source references on 2026-09-13. Explanations, scenarios, and diagrams are original teaching material. Other releases and managed runtimes can differ.