What is Apache Spark?

Why distributed processing?

Imagine computing daily revenue from a year of transaction events. On a laptop, the data may exceed available memory or take longer than your processing window. A distributed engine coordinates work across machines. That coordination has a cost: a small CSV may be simpler and faster to process with pandas or a database.

The useful interview question is: what limits this workload—data size, CPU, I/O, or latency? “Big data” alone is not a sizing decision.

Compute is different from storage

Spark reads data, computes results, and writes them to a destination. Your source might be object storage or a database; Spark itself is not your permanent database. Storage, table formats, and compute are separate choices.

A small example, a useful habit

Start with a tiny dataset whose result you can calculate yourself. Here, two successful payments total ₹1,200. The same expression can later operate over partitioned production data. This example assumes a spark session; the next lesson explains its architecture.

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F
payments = spark.createDataFrame(
    [("success", 500), ("failed", 200), ("success", 700)],
    "status string, amount long"
)
revenue = (payments.filter(F.col("status") == "success")
           .agg(F.sum("amount").alias("revenue")))
revenue.show()

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

Would you use Spark to process a 20 MB CSV every morning?

Say your answer out loud before opening the notes.