Joins & data skew
Reason about data shape before reaching for a join hint.
Choose a join strategy using data sizes, key distribution, and the actual execution plan—not table names.
Start with correctness
Identify the join keys, join type, null behavior, and expected row count. Duplicate keys can multiply output rows. A fast join that changes business grain is still wrong.
Small side, large side
A broadcast join distributes a build-side relation to executors. It can avoid shuffling the large side for a suitable join, but the broadcast must fit safely. Filter and project the small relation before assessing size. Hints are requests with strategy limitations, not guarantees.
When one key dominates
If most events share one customer key, hashing that key still sends its records together. Diagnose task and partition imbalance. Possible responses include isolating hot keys, pre-aggregation where valid, or carefully designed salting. Check correctness when combining salted results.
Try it on a small dataset
Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.
from pyspark.sql import functions as F
facts = spark.createDataFrame([(1, 100), (2, 200)],
"customer_id long, amount long")
customers = spark.createDataFrame([(1, "Pune"), (2, "Delhi")],
"customer_id long, city string")
joined = facts.join(F.broadcast(customers), "customer_id")
joined.explain("formatted")
joined.orderBy("customer_id").show()Two matching rows: customer 1 / 100 / Pune and customer 2 / 200 / Delhi.
Result reasoned through on the small input; not runtime-tested against a Spark cluster.
Do not broadcast a relation simply because it is smaller than the other table. Its expanded size and executor memory pressure matter.
A join finishes 199 tasks quickly and one task takes 20 minutes. What do you inspect?
Say your answer out loud before opening the notes.
Reveal the reasoning
Compare the slow task’s input and shuffle size, spill, and execution time. Check hot or null keys and join multiplicity. Consider adaptive skew handling where applicable; do not assume that adding executors splits a single hot partition.
Sources & verification
Core claims checked against Apache Spark 3.5.7 documentation and source references on 2026-09-13. Explanations, scenarios, and diagrams are original teaching material. Other releases and managed runtimes can differ.