Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

What is a DAG? What is Catalyst?

Mediumdagcatalystoptimizerexecution

Interview question

In Spark, what is a DAG, and what is the Catalyst optimizer?

How to answer

The short answer

(Directed Acyclic Graph): when you chain transformations, Spark builds a DAG of stages and tasks. Nodes are operations; edges are dependencies. "Acyclic" means lineage flows one way so Spark can recompute lost .

Output
scan parquet --> filter --> project --> exchange(shuffle) --> aggregate --> write
                 stage 1                         stage 2

Stages split at boundaries (wide dependencies).

Catalyst

Catalyst is Spark SQL's . For /Dataset/SQL plans it:

  1. 1Analyzes the logical plan (resolve columns, types)
  2. 2Applies rule-based optimizations (, constant folding, projection pruning)
  3. 3Builds a cost-based physical plan (join strategy selection when stats exist)
  4. 4Generates Java bytecode (Tungsten / whole-stage codegen) for many operators
Output
SQL / DataFrame API
        |
   Unresolved Logical Plan
        |
   Analyzed Logical Plan
        |
   Optimized Logical Plan   <--- Catalyst rules
        |
   Physical Plan(s)
        |
   Selected Physical Plan
        |
   Codegen + Execution

See the plan in code

Python
df = (
    spark.read.parquet("/data/orders")
    .filter("amount > 100")
    .select("order_id", "amount")
)
df.explain(True)          # parsed / analyzed / optimized / physical
df.explain("formatted")

What to say

DataFrames get Catalyst optimizations. Raw RDD map pipelines usually do not get the same SQL-style plan rewrites.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill