Partitions & shuffles
Follow the data when work needs to cross partition boundaries.
A shuffle redistributes records across partitions. It is often necessary when records sharing a key must meet.
Local work versus redistribution
A filter can evaluate rows where they already are. A grouped aggregation needs values for each key brought together. Spark can sometimes reduce data locally first, but the remaining partial results still need appropriate distribution.
Repartition or coalesce?
repartition redistributes data through a shuffle. coalesce can reduce the partition count without a shuffle, but a drastic reduction can constrain parallelism. Neither operation is a universal fix for output file sizing.
Partitioning has several meanings
An execution partition is not the same as a storage directory such as date=2026-09-13. Input files, execution partitions, shuffle partitions, and output files are related through execution and writer behavior; they are not interchangeable counts.
Try it on a small dataset
Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.
from pyspark.sql import functions as F
rows = spark.range(0, 12, 1, 3)
keyed = rows.withColumn("key", F.col("id") % 2)
redistributed = keyed.repartition(4, "key")
redistributed.explain("formatted")Look for an Exchange with hash partitioning on key. Four requested partitions do not guarantee four nonempty partitions.
Result reasoned through on the small input; not runtime-tested against a Spark cluster.
More partitions can improve parallelism, but too many tiny tasks add overhead. A hot key can remain hot after repartitioning by that same key.
Why did coalesce(1) make my write slow?
Say your answer out loud before opening the notes.
Reveal the reasoning
It funnels output through one partition and can reduce upstream parallelism. Check file requirements; prefer a useful number of output partitions. A shuffle before the final write trades extra redistribution for parallel upstream work.
Sources & verification
Core claims checked against Apache Spark 3.5.7 documentation and source references on 2026-09-13. Explanations, scenarios, and diagrams are original teaching material. Other releases and managed runtimes can differ.