Caching & fault tolerance

Cache a reusable boundary

If several actions reuse an expensive filtered dataset, persisting that intermediate result may help. Materialization costs time and storage. Compare total runtime, check the Storage tab, and release data when its reuse window ends.

Be precise about defaults

In PySpark 3.5.7, DataFrame cache() uses MEMORY_AND_DISK_DESER. Do not transfer this default to every RDD API or version. Storage settings and eviction determine what remains available for reuse.

Recovery has boundaries

Spark can recompute lost RDD partitions using lineage. That requires usable upstream inputs; it does not promise that every external side effect runs exactly once. Retry-aware writes, deterministic logic, and appropriate transactional sinks are separate design decisions.

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F
reusable = spark.range(100).filter(F.col("id") % 2 == 0).cache()
try:
    print(reusable.count())  # materializes this small example
    reusable.agg(F.sum("id").alias("total")).show()
finally:
    reusable.unpersist()

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

Is caching every DataFrame a good way to speed up my pipeline?

Say your answer out loud before opening the notes.