RDDs, DataFrames & Spark SQL

A schema is part of the contract

A DataFrame is organized into named, typed columns. Define the expected schema when ingestion needs to be predictable. Check nullability, malformed records, and type conversions explicitly instead of assuming that successful reading means clean data.

Where RDDs fit

RDDs expose lower-level distributed collections. They are useful for custom processing that does not fit structured expressions. Python DataFrames do not provide the Scala/Java typed Dataset API. In an interview, separate the concepts from language-specific API names.

SQL or DataFrame?

These are two ways of expressing structured computation through Spark SQL. Choose the form your team can maintain, then inspect the plan. Equivalent logic may produce equivalent plans; switching syntax alone is not a performance strategy.

Try it on a small dataset

Run this in a PySpark notebook with an existing spark session. Create a local session if needed. Examples are illustrative; this website does not execute Spark.

orders = spark.createDataFrame(
    [("Pune", 500), ("Delhi", 700), ("Pune", 300)],
    "city string, amount long"
)
orders.createOrReplaceTempView("orders")
spark.sql("""
    SELECT city, SUM(amount) AS revenue
    FROM orders GROUP BY city ORDER BY city
""").show()

Result reasoned through on the small input; not runtime-tested against a Spark cluster.

Is Spark SQL slower than the DataFrame API?

Say your answer out loud before opening the notes.