Jobs, stages & tasks

Work at three scales

A job is work initiated to obtain a result. A stage groups tasks that can run without crossing a shuffle dependency. Within a stage, tasks operate on partitions. Task attempts can be retried, so attempts and distinct tasks are different counts.

Partitions are not CPU cores

In the simplified case of 12 ready tasks and four available task slots, execution needs at least three waves. That assumes one slot per task, similar task lengths, and no failures. Twelve partitions do not imply twelve machines.

Read the UI from the slow end

Find the slow stage, compare task durations, and inspect input sizes and shuffle metrics. One unusually slow task suggests a different investigation from every task being slow. Use evidence to distinguish imbalance from insufficient overall resources.

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F
events = spark.range(0, 120, 1, 4)
counts = (events.withColumn("bucket", F.col("id") % 3)
          .groupBy("bucket").count())
counts.explain("formatted")
counts.orderBy("bucket").show()

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

You have 200 tasks and 20 available slots. Will the stage take exactly ten task durations?

Say your answer out loud before opening the notes.