Big ideas.
Smaller partitions.
A place to rebuild your Spark fundamentals, follow the data,
and walk into your next interview with a clearer head.
01 — DIVIDE THE DATA. SHARE THE WORK.First, get the foundations right.
Follow the sequence, or pick up the concept you need today.
01 / THE FUNDAMENTALS
01What is Apache Spark?
Start with the problem Spark solves, before the API you write.
Driver, executors & the cluster
Know who plans the work and who actually runs it.
RDDs, DataFrames & Spark SQL
Choose the right abstraction for a data engineering pipeline.
Lazy evaluation & actions
Understand why a line of code is not a job.
Jobs, stages & tasks
Read the execution hierarchy without memorizing misleading formulas.
Partitions & shuffles
Follow the data when work needs to cross partition boundaries.
Joins & data skew
Reason about data shape before reaching for a join hint.
Caching & fault tolerance
Reuse the right work, and know what Spark can recompute.
Executor memory, spill & out-of-memory errors
Understand where an executor's memory goes, why Spark spills to disk, and how to read an out-of-memory failure.
Reading the Spark UI and fixing slow jobs
Start from the slowest stage, compare tasks, then read the SQL plan before changing anything.
Catalyst, code generation & AQE
Follow a DataFrame from parsed plan to executed plan, read what explain prints, and see what adaptive execution changes once real sizes arrive.
PySpark UDFs, Arrow & the Python boundary
See where your Python function actually runs, what crosses into it, and why the fastest UDF is usually the one you did not need to write.
Storage formats: Parquet, ORC & Avro
Why the file layout decides what your query reads — and why small files are fatal.
02 / TABLES & STREAMS
14Delta Lake: transactions, MERGE & maintenance
How an ordered log turns immutable Parquet files into a table, and what MERGE, OPTIMIZE, and VACUUM really cost.
Structured Streaming: micro-batches, state & exactly-once
What the driver plans each trigger, what the checkpoint records, and why late data and restarts behave the way they do.
03 / CONNECT & APPLY
16The complete Spark execution flow
From spark-submit to the final result: explain who does what, when, and why.
Predict what Spark will do
Three interview exercises. Read the code, predict the work, then check your reasoning against the plan and Spark UI.
Count the jobs, stages, and tasks
Eight practice snippets measured with the Spark UI REST API, with AQE on and off, so you can give counts along with the assumptions that make them true.
The PySpark coding round: six patterns
Six problems that recur in 30–45 minute data engineering coding rounds — each solved, run, and costed, with the wrong answer that looks right.
Designing data pipelines: the senior round
A repeatable answer shape, and the idempotent, ordered, recoverable write paths that make a design survive a rerun.