Executor memory, spill & out-of-memory errors

Where the memory goes

An executor is a JVM process plus whatever runs beside it. spark.executor.memory sizes the JVM heap. spark.executor.memoryOverhead budgets additional memory for VM overheads, native allocations, and Python workers when spark.executor.pyspark.memory is unset. If overhead is not set explicitly, Spark uses spark.executor.memoryOverheadFactor with a 384 MiB minimum: the factor defaults to 0.10, except for Kubernetes non-JVM jobs such as PySpark, where it defaults to 0.40.

On YARN and Kubernetes, the executor container budget is the sum of spark.executor.memory, spark.executor.memoryOverhead, enabled off-heap memory (spark.memory.offHeap.size), and separately configured spark.executor.pyspark.memory. The optional allocations add to the container request; Python workers share the overhead budget when no separate PySpark allocation is configured.

Inside the heap, Spark reserves 300 MiB for system use, then splits what remains. spark.memory.fraction (default 0.6) is the region shared by execution and storage memory; spark.memory.storageFraction (default 0.5) marks how much of that region cached blocks can claim without being evicted.

• Execution memory holds working state: shuffle buffers, sort runs, and hash tables for joins and aggregations.
• Storage memory holds cached blocks for reuse.
• Execution may evict cached storage when it needs space. Storage may never evict execution — the source comments call this out as a deliberate simplification.

Two consequences matter in interviews: a pandas conversion or a large broadcast lands somewhere specific — driver or executor — and a cache that seems to disappear under heavy compute is the eviction rule at work, not a bug.

Check the sources: Executor memory configuration · Memory tuning overview · UnifiedMemoryManager

What spills, and what it costs

Operators that need to see many rows at once — sorts, hash aggregations, shuffle writers — accumulate state in execution memory. When the memory manager cannot grant more, spillable structures write sorted runs to local disk and merge them later. The sort path carries an explicit element threshold that forces a spill before the structure grows arbitrarily, and spilled files land in Spark's configured local scratch directories, not remote storage.

Spill is graceful, not fatal: the stage completes more slowly because it traded memory pressure for disk I/O. It deserves action when it is large relative to the stage's input, or when it concentrates in a few tasks — compare tasks with each other instead of expecting any fixed size.

The Spark UI reports it directly. The Stages tab lists Shuffle spill (memory) — the deserialized form of the shuffled data held in memory — and Shuffle spill (disk) — the serialized form written to disk — next to peak execution memory and GC time. The SQL tab attributes spill size and peak memory to specific operators such as Sort and HashAggregate.

Check the sources: Web UI metrics · UnsafeExternalSorter

Three out-of-memory failures, three places to look

Driver OOM. The driver collects or materializes too much: collect(), toPandas(), a huge returned result, or a giant broadcast. spark.driver.maxResultSize exists to abort a job before this happens; the limit lives in the driver's heap, not an executor's.

Executor task OOM. The working set of one task is too large — one oversized partition, or one hot key carrying a disproportionate share of rows. This is the failure the tuning guide describes for reduce-side operations, and its first suggested fix is more parallelism, so each task's input set is smaller. Adding executors does not split the offending partition.

Container or overhead kill. The process is killed by YARN or Kubernetes for exceeding the container's memory limit while the JVM heap looks healthy. What ran out is non-heap memory: overhead, off-heap allocations, or a PySpark worker — Python memory lives outside the JVM heap and counts against overhead unless spark.executor.pyspark.memory sets its own limit.

Read the failure location, not just the exception text: driver logs, executor stderr, exit codes, and the Spark UI's per-task view each point at a different one of these.

Check the sources: Configuration reference · Memory tuning overview

How to debug it, in order

Work from observation to action, and keep the unit of work in mind at every step:

1. Locate the failure. Driver log, executor stderr, or a killed container. Find the stage and the tasks that failed or slowed first.
2. Classify the pressure. Spill and GC high across every task suggests general memory pressure; one task far above the median suggests skew or an oversized partition.
3. Shrink the unit of work. More or smaller partitions make each task's working set smaller; avoid collecting results to the driver, and fix skew where the data shape allows it.
4. Check memory hygiene. Unpersist what is no longer reused — caching everything competes with execution memory.
5. Only then change memory sizes. Raise executor memory when the single unit genuinely needs it; raise overhead when containers were killed despite a healthy heap; treat Python worker memory separately for PySpark.

Interviewers listen for exactly this structure: where the failure happened, which unit could not fit, and why the chosen fix changes that unit's size — instead of "add more executors" applied blindly.

Check the sources: Memory tuning overview · Configuration reference

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F

orders = spark.read.parquet("orders")  # customer_id, amount

totals = (
    orders
    .groupBy("customer_id")
    .agg(F.sum("amount").alias("total"))
)

totals.write.mode("overwrite").parquet("customer_totals")

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

A groupBy job fails with OutOfMemoryError. How do you decide whether to change the data layout, the code, or the executor configuration?

Say your answer out loud before opening the notes.