Lazy evaluation & actions

Build a plan, then execute

Think of a transaction pipeline as a recipe: read, filter, select, aggregate. Describing a recipe is different from cooking it. Deferring execution lets Spark inspect the requested computation before running it.

Predict before you run

In this example, building even describes a filter. count() asks for a result. A second action can require work again unless an appropriate intermediate result has been persisted and materialized.

Use the plan as evidence

explain("formatted") helps you inspect execution operators without collecting the dataset. Distinguish planning from execution, and avoid claiming that every API call is free: schema inference and metadata operations can do work too.

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F
numbers = spark.range(0, 10)
even = numbers.filter(F.col("id") % 2 == 0)
even.explain("formatted")
print(even.count())

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

I wrote five transformations. How many jobs did I create?

Say your answer out loud before opening the notes.