The complete Spark execution flow

Before you draw the flow

Open with a scope statement: “I’ll describe a classic PySpark batch application using DataFrames on Spark 3.5.7.” Then separate the application’s lifetime from the execution of an individual query. One application can run many queries and jobs.

This matters in interviews: a notebook may already have a session and compute attached. With Spark Connect, a client sends plans to a remote server; the Python client is not the classic driver described below. Structured Streaming adds a continuously running query and its own execution lifecycle.

Learn the fundamentals first if the terms are new. This is the capstone that connects them.

Check the source: Spark Connect has a separate client/server architecture · Application components and resource allocation

1–4 · Start the application

1. Submit the program. A command such as spark-submit app.py launches the application using its master and deployment configuration. In client mode the driver runs on the submitting side; in cluster mode the cluster infrastructure launches it. The deployment mode determines driver placement.
2. Initialize or reuse the session. Driver-side code obtains a SparkSession. In classic Spark, it uses a SparkContext for the application’s connection and scheduling machinery. getOrCreate() may return an existing session; it is not a command that always launches a fresh driver.
3. Acquire executor resources. Cluster-manager-specific integration requests resources. YARN, Kubernetes, and Spark standalone are common choices; Mesos is deprecated in this reference release. Executors are processes, not synonyms for worker machines. Memory requests also have overhead beyond the configured executor heap.
4. Register and become schedulable. Executors connect back to the driver. Resource availability and readiness settings affect when scheduling begins. There is no universal rule that every requested executor must exist before work can start. Dynamic allocation can add or remove executors later.

In local mode, task threads run locally; there is no remote cluster-manager allocation of three separate executors. In an already attached notebook, much of this startup may have happened before your cell ran.

Check the source: Driver placement and spark-submit · SparkSession.getOrCreate: create or reuse · Application components and resource allocation · Scheduling and dynamic resource allocation · Task CPUs, registration readiness, and retry configuration

5–8 · Describe the query and plan it

5. Run the driver-side program. Python control flow executes normally. Reading a table and building expressions such as filter and groupBy describe a distributed computation. Metadata discovery, schema inference, and analysis can already do work or report errors.
6. Analyze the logical plan. Spark SQL resolves references to columns, relations, functions, and types. “Aggregate over a filter over an input relation” describes the requested result. A logical input relation is not yet a physical FileScan operator.
7. Optimize the logical plan. Applicable rules simplify expressions and reduce unnecessary work. Column pruning and eligible filter pushdown are useful examples. Pushdown depends on the source and predicate; a remaining filter may still need to run. Join reordering and strategy choices have eligibility, statistics, and configuration requirements.
8. Choose and prepare physical operators. The physical plan specifies how to compute the result: scans, filters, partial/final aggregates, joins, and exchanges. Eligible operators can be fused into generated code. That fusion is distinct from a scheduler stage.

Catalyst plans structured queries. The DAG scheduler later works with execution dependencies; it does not schedule the Catalyst logical tree directly. Ordinary RDD transformations do not go through the same DataFrame/SQL optimization pipeline.

Check the source: QueryExecution: analysis, optimization, physical planning · Reading EXPLAIN output · Physical strategy selection and join eligibility

9–11 · Request results and schedule work

9. Call an action. show(), count(), collect(), or an actual write operation asks for an outcome. Accessing df.write alone only creates a writer. Necessary planning is completed and execution work is submitted. One user action can lead to multiple scheduler jobs; query execution and job are different levels.
10. Build the stage dependencies and tasks. The DAG scheduler identifies shuffle dependencies and available intermediate results. For a runnable stage attempt, it creates the required tasks and submits them as a TaskSet. A ShuffleMapStage produces shuffle output; a ResultStage produces the result requested by that job.
11. Place task attempts on executors. The task scheduler uses resource offers, locality preferences, scheduling policy, and task resource requirements. The scheduler backend dispatches the selected work to executors. The cluster manager supplies resources; it is not an obligatory relay for each task or each data row.

Say “one task per partition to compute in that stage attempt”, not “one task per original file.” Some actions need only a subset of partitions. Retries and speculation create additional attempts, and downstream partition counts can change.

This is a capacity model, not a timing promise. Other resources, competing jobs, skew, and task durations matter. Tasks run on executor threads; Spark does not permanently pin a partition to a physical CPU core.

Check the source: Actions, lazy transformations, and recomputation · DAGScheduler: stages, tasks, TaskSets, and recovery · TaskSchedulerImpl: task placement and resource offers · Task CPUs, registration readiness, and retry configuration

12–14 · Process partitions and shuffle

12. Execute the task’s computation. An executor deserializes task information and processes the assigned execution partition. Pipelined operators can read, filter, project, and partly aggregate within one task. Built-in SQL expressions run through Spark’s execution engine; Python UDFs can involve Python worker processes.
13. Produce and fetch shuffle blocks when required. For a grouped sum, map-side tasks may first produce partial sums. A shuffle exchange distributes these by key. The same key must reach compatible downstream grouping, but multiple distinct keys can share a hash partition. Downstream tasks fetch the required blocks, locally or remotely.
14. Run downstream operators; adapt when applicable. Once dependencies are available, downstream tasks complete the aggregation. With AQE enabled, materialized exchange statistics can inform changes such as coalescing shuffle partitions or selecting a different supported join strategy. Initial and final plans can differ.

Shuffle can involve serialization, disk I/O, network I/O, and spill. A network transfer is not required for every block. BroadcastExchange instead prepares and distributes a broadcast relation; treating every operator containing “Exchange” as a shuffle is inaccurate.

Locality tries to avoid avoidable movement. PROCESS_LOCAL, NODE_LOCAL, RACK_LOCAL, and ANY describe locality choices; NO_PREF means there is no locality preference. Object-storage reads do not imply that an executor lives beside an HDFS block.

Check the source: Shuffle dependencies and output partitioning · BroadcastExchange is a different exchange · Adaptive query execution and shuffle partition coalescing · Data locality and its trade-offs

An AQE query stage is a physical-plan unit around an exchange; a whole-stage codegen region is an operator-fusion unit. Neither is interchangeable with a DAG scheduler stage in the Jobs/Stages UI. See the AQE query-stage implementation.

15–17 · Observe, recover, and finish

15. Track progress throughout execution. Task status updates and metrics feed the driver’s view of the application. Use the SQL tab for query plans and linked jobs; use stage/task views for skew, shuffle, spill, duration, and GC evidence. The Storage tab concerns materialized persisted data.
16. Recover eligible failures. A failed task attempt may be retried within configured limits. Lost shuffle output can require recomputation of earlier work. Recovery needs available inputs and recomputable logic; a deterministic code error will not be repaired by simply retrying it. Driver failure is a different lifecycle problem, not a task retry.
17. Return or commit the output. show() brings back enough rows to display, although computing those rows can require substantial upstream work. collect() returns all result rows to the driver. Distributed file writes normally write data from executors and coordinate commits; output data need not pass through the driver. One executor does not imply one output file. The application can accept another action, or terminate and release resources.

Do not claim that external side effects happen exactly once merely because Spark retries tasks. Side-effect safety and a sink’s commit guarantees need their own design. Do not stop a shared notebook session just because one query has finished.

Check the source: SQL, Jobs, Stages, Executors, and Storage tabs · DAGScheduler: stages, tasks, TaskSets, and recovery · Actions, lazy transformations, and recomputation

Trace the example before running it

The runnable example below creates 12 rows in four input partitions. Each ID determines a country and an amount. Filtering amount > 100 keeps IDs 3–11. Here is the input-side arithmetic:

In a typical full-input execution, each input partition needs a task, even P0: Spark must evaluate its rows before knowing none survive. Partial sums can be shuffled and merged to produce India 900, USA 1050, UK 1200. The output order is unspecified.

Range (4 input partitions)
  → Filter amount > 100
  → Partial aggregate by country
  → Shuffle exchange by country
  → Final aggregate
  → Display a small result

This sketches the aggregation’s shuffle boundary, not an assertion that show() must produce exactly one job and two stages. AQE and result retrieval influence the observed scheduler work. spark.sql.shuffle.partitions does not define the four input partitions in this example.

For your Parquet version, replace the generated input with spark.read.parquet("/orders"). Inspect the actual scan and partitioning instead of assuming one input file means one task. Look for pushed filters and selected columns; do not expect a literal operator-for-operator match with this range example.

Check the source: Reading EXPLAIN output · Shuffle dependencies and output partitioning · Adaptive query execution and shuffle partition coalescing

A repeatable way to answer in an interview

1. Scope it. State classic PySpark or Spark Connect, batch or streaming, and whether the application is already running.
2. Draw the roles. Put the driver beside the cluster manager and executors. Show resource allocation separately from task dispatch and data access.
3. Walk the query. Use one scan → filter → grouped sum → action example. Identify what is a plan and what actually processes data.
4. Explain the expensive boundary. Draw the partial aggregate, shuffle write/read, and final aggregate. Say what crosses the boundary.
5. Size the work with assumptions. Separate input partitions, shuffle partitions, task attempts, and concurrent slots. Then discuss skew and retries.
6. Verify the story. Name the plan operator or Spark UI evidence you would inspect. Finish with where the result goes and whether the application stays alive.

Practice at two depths: give the answer below in roughly 90 seconds, then walk through the example when the interviewer asks for detail. Avoid reciting 17 steps without explaining who owns each step.

Check the source: SQL, Jobs, Stages, Executors, and Storage tabs

Corrections to the supplied explanation

The supplied answer is a useful starting outline. These changes make it safer to use as an interview explanation:

Keep the relationships, but say them precisely: action requests results; shuffle dependencies separate scheduler stages; tasks compute stage partitions; resource slots bound concurrency; executors run task attempts.

Check the source: SparkSession.getOrCreate: create or reuse · QueryExecution: analysis, optimization, physical planning · DAGScheduler: stages, tasks, TaskSets, and recovery · TaskSchedulerImpl: task placement and resource offers · BroadcastExchange is a different exchange · Physical strategy selection and join eligibility · Task CPUs, registration readiness, and retry configuration · Adaptive query execution and shuffle partition coalescing

Further checks: Window distribution requirements · Incremental result retrieval

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

from pyspark.sql import functions as F

# Uses an existing classic SparkSession named spark.
orders = (
    spark.range(0, 12, 1, 4)
    .withColumn("country",
        F.when(F.col("id") % 3 == 0, "India")
         .when(F.col("id") % 3 == 1, "USA")
         .otherwise("UK"))
    .withColumn("amount", F.col("id") * 50)
)
result = (
    orders.filter(F.col("amount") > 100)
    .groupBy("country")
    .agg(F.sum("amount").alias("total"))
)

result.explain("extended")  # inspect plans, not result rows
result.show(truncate=False) # request the result

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

Can you explain the full Spark flow, from application startup to the final result?

Say your answer out loud before opening the notes.