Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Core Concepts

What is Spark UI?

Mediumspark-uidebuggingobservability

Interview question

What is the Spark UI, and what do you look at when debugging a slow job?

How to answer

The short answer

The Spark UI is Spark's web dashboard for an application (typically port 4040 on the driver, or via YARN/History Server for finished apps).

Key tabs

What I check first on a slow job

  1. 1SQL tab: physical plan, whether broadcast was used, filters.
  2. 2Stages: longest stage, whether it is a stage.
  3. 3Tasks: skewed max/median duration ( signal).
  4. 4Shuffle read/write and spill (memory/disk).
  5. 5GC time and executor lost / speculation.

Practical workflow

Output
slow action
  -> find Job
  -> find longest Stage
  -> inspect straggler tasks
  -> relate back to join/agg key and input size

Tip in code / local runs

Python
# UI usually at http://localhost:4040 while the app is alive
spark = SparkSession.builder.appName("debug-me").getOrCreate()
df.groupBy("user_id").count().write.mode("overwrite").parquet("/tmp/out")
# Inspect Jobs/Stages/SQL tabs while this action runs (or use History Server after)

Interview tip

Mention History Server for post-mortem of production jobs. Reading the plan beats guessing config knobs.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill