Reading the Spark UI and fixing slow jobs

Start from the slow end

Open the Jobs tab first: it lists every job with status, duration, and a tasks progress bar, plus an event timeline of executors and jobs. Click the longest-running job to reach its detail page — event timeline, DAG visualization, and its stages grouped by state (active, pending, completed, skipped, failed).

From there, open the Stages tab and sort by duration. Each stage row shows how far its tasks progressed, how much it read from storage, and how much it shuffled: shuffle read (bytes and records read, local plus remote) and shuffle write (bytes and records written for a later stage to read). The slowest stage is your investigation target — not the first stage, and not an average over the whole application.

Two checks before drilling in: the Executors tab tells you whether executors were even available (memory, cores, task and shuffle counts, GC time), and the Environment tab shows explicitly supplied SparkContext properties, including startup memory settings; default values are not all listed. Session SQL changes made with spark.conf.set(...) or SQL SET are not reflected there. Before running the query, read the current session values with spark.conf.get("spark.sql.shuffle.partitions") and spark.conf.get("spark.sql.adaptive.enabled"), or SQL SET, then inspect the executed plan. Debugging a slow job with the wrong assumed config wastes the whole pass.

Check the sources: Web UI: Jobs, Stages, Executors, Environment tabs · Startup properties and runtime SQL configuration

Jobs, stages, and the task distribution

A job is a triggered computation; a stage is the set of tasks that can run without redistributing data; a task is one partition processed in one attempt. The stage detail page makes this concrete: total time across all tasks, locality-level summary, shuffle read size and records, associated job IDs, and a DAG whose nodes are grouped by operation scope (BatchScan, WholeStageCodegen, Exchange).

Read the summary metrics table, not just the duration: task duration, GC time, result serialization and result-fetch time, scheduler delay, peak execution memory, shuffle read size and records, shuffle fetch wait time (tasks blocked waiting on remote shuffle data), remote reads, shuffle write time, and shuffle spill (memory) versus (disk) — the deserialized bytes held in memory and the serialized bytes written to disk. The same metrics aggregated by executor show whether one executor carried the load; the per-task table shows duration, input, shuffle, spill, and GC per task, with logs and attempt numbers for failures.

• Every task slow together suggests general pressure: too little parallelism, oversized partitions, heavy spill and GC everywhere, or data fetched from far away (compare PROCESS_LOCAL and NODE_LOCAL against RACK_LOCAL and ANY).
• One task far above the median suggests skew or one oversized partition: compare its input and shuffle size, spill, and duration against the median task instead of trusting the stage average.

The tuning guide's first fix for an oversized reduce-side working set is more parallelism so each task's input set is smaller — Spark reuses one executor JVM across many tasks and supports tasks as short as 200 ms. Recommend 2–3 tasks per CPU core as a starting point, not a guarantee.

Check the sources: Web UI: stage and task metrics · Tuning: parallelism, reduce memory, locality

Read the SQL plan with the stages

The SQL tab lists each executed query with duration, associated jobs, and its physical and logical plans. Its detail page shows the query DAG — WholeStageCodegen blocks (with codegen IDs such as WholeStageCodegen (1)) fusing scans, filters, and partial aggregates into one function, and Exchange blocks carrying shuffle metrics such as records written and data size. A Details link exposes the parsed, analyzed, optimized, and physical plans; operators subject to whole-stage codegen are prefixed with a star and codegen ID.

Each operator block reports SQL metrics: number of output rows after a filter, aggregate, or join; shuffle bytes written and write time on an Exchange; spill size and peak memory on Sort and HashAggregate; scan and metadata time on file scans; build-side size and hash-map build time on joins. Use these to name the expensive operator: a Sort with large spill size is an execution-memory problem; an Exchange with huge remote reads plus high fetch wait is a redistribution problem; a join whose build side dwarfs the threshold is a strategy problem.

Cross-reference both directions: stage DAG nodes carry scope names and codegen IDs that map back to the SQL plan, and the SQL page links each query to its jobs. The Storage tab completes the picture for reuse — persisted DataFrames appear only after an action materializes them, with storage level, size, and partition counts. A cache that never appears was never materialized; a cache that shrinks under compute lost to execution eviction.

Check the sources: Web UI: SQL metrics and Storage tab

Fix in order: layout, code, then config

Work from observation to the smallest change that shrinks the largest unit of work:

1. Shrink or rebalance the unit of work. More or smaller partitions for oversized reduce inputs; avoid collect() and toPandas() on large results; fix hot or null keys where the data shape allows it. Adding executors does not split one hot partition.
2. Check reuse hygiene. Cache only repeatedly read inputs, confirm materialization in the Storage tab, and unpersist what is no longer reused — cached blocks compete with execution memory.
3. Let the optimizer help where eligible. Adaptive query execution is enabled by default since Spark 3.2 under spark.sql.adaptive.enabled: it coalesces post-shuffle partitions from a large initial count, converts sort-merge joins to broadcast joins when a side proves small at runtime, and splits skewed sort-merge join partitions. Each has eligibility and threshold requirements — read the executed plan rather than assuming AQE removed the shuffle.
4. Change configuration last, for the component that died. spark.sql.shuffle.partitions for shuffle parallelism, AQE size thresholds for coalescing and skew handling, broadcast thresholds for join strategy, memory settings for genuine capacity shortfalls. Confirm startup Spark properties in the Environment tab. For session SQL changes, use spark.conf.get(...) or SQL SET in the session that will run the query. Re-run it and inspect the executed plan and stage metrics to verify the effect.

Interviewers listen for this sequence: slowest stage first, task distribution second, SQL operator third, fix last — with the UI screen named at each step instead of "tune Spark" applied blindly.

Check the sources: SQL tuning: shuffle partitions and AQE · Tuning: parallelism and reduce memory · Startup properties and runtime SQL configuration

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F

orders = spark.read.parquet("orders")  # order_id, customer_id, amount

totals = (
    orders
    .repartition(200, "customer_id")
    .groupBy("customer_id")
    .agg(F.sum("amount").alias("total"))
)

totals.write.mode("overwrite").parquet("customer_totals")

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

A stage has 199 fast tasks and one task that takes 20 minutes. Where do you look first, and what do you change last?

Say your answer out loud before opening the notes.