Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

RDD vs DataFrame vs Dataset

Mediumrdddataframedatasetapi

Interview question

Compare RDD, DataFrame, and Dataset in Spark. When would you use each?

How to answer

The short answer

Spark has three primary abstractions. In PySpark, you mostly use . Datasets are a Scala/Java concept (typed API); Python does not expose Dataset the same way.

Comparison

Output
RDD (Resilient Distributed Dataset)
  - typed as Python objects / JVM objects
  - low-level transformations (map, flatMap, reduceByKey)
  - little catalyst optimization
  - use for custom partition logic or legacy code

DataFrame
  - rows with a schema (like a distributed table)
  - Catalyst + Tungsten optimized
  - Spark SQL, column expressions, joins, windows
  - default choice in PySpark

Dataset[T] (Scala/Java)
  - typed records + encoder
  - compile-time type safety with Catalyst
  - not the day-to-day PySpark surface

Example

Python
# DataFrame (preferred)
df = spark.read.parquet("/data/events")
df.filter(df.event == "purchase").groupBy("user_id").count()

# RDD (when you truly need it)
rdd = df.rdd.map(lambda row: (row.user_id, 1))
counts = rdd.reduceByKey(lambda a, b: a + b)

When to use what

  • DataFrame: almost always (, analytics, streaming structured APIs).
  • RDD: rare edge cases (custom partitioning, non-relational algorithms, migrating old code).
  • Prefer built-in column functions over RDD map for speed and whole-stage codegen.

Common interview answer

"DataFrames sit on RDDs under the hood, but the declarative API lets Catalyst reorder, prune, and codegen the plan. I use DataFrames unless I hit a limitation that only RDDs solve."

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill