What actually moves
during a shuffle?
Follow six rows from scattered input to groups that share a key.
Input: each partition contains a mix of cities. Pune appears in all three partitions.
A conceptual model, not a Spark execution or performance benchmark. Real aggregations may combine locally before a shuffle; hash partitions can contain multiple keys, and AQE can change partition counts.
Follow the key
Grouping requires compatible distribution: all partial results for a city must reach the right downstream partition. Here, we illustrate a simplified assignment with one city per output partition.
Compare with a filter
A filter evaluates rows in their existing partitions. Select “Filter to Pune” to see rows disappear without crossing partition boundaries.
One shuffle, frame by frame.
Read the full video transcript
0–3 seconds: Six rows are split across three input partitions. Each row has a city key.
3–8 seconds: Rows move to the illustrated output partition for their city. Pune rows come together; Delhi rows come together.
8–12 seconds: The grouped result has Pune: 3, Delhi: 2, and Mumbai: 1. This illustrates redistribution; real Spark can partially aggregate before moving data.
Technical reference: Apache Spark shuffle operations ↗