Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Partitioning & Pruning

What is a Spark stage?

Mediumstagedagexecutionspark-ui

Interview question

What is a Spark stage?

How to answer

The short answer

A stage is a set of tasks that can run together without a boundary. Spark splits a job's into stages at wide dependencies.

Hierarchy

Output
Application
  -> Jobs          (one per action)
     -> Stages     (pipelined narrow transforms)
        -> Tasks   (one per partition)

Diagram

Output
Stage 0: scan -> filter -> project
                 \
                  shuffle exchange
                 /
Stage 1: aggregate -> project -> write

Why that matters

  • The Spark UI's longest stage is usually where you optimize first
  • Task count in a stage ~= number of for that stage
  • Stragglers inside a stage often mean skew or uneven file sizes
Python
# Each action triggers a job with one or more stages
df.filter("amount > 0").groupBy("user_id").sum("amount").write.parquet("/out")
# typically: stage for map-side prep + stage for post-shuffle aggregation + write

Interview tip

"Jobs are triggered by actions; stages are separated by shuffles; tasks run per partition." That sentence alone answers a lot of Spark interview questions.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill