Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Joins & Performance

What is AQE?

Hardaqeoptimizerspark3performance

Interview question

What is Adaptive Query Execution (AQE) in Spark?

How to answer

The short answer

(AQE) lets Spark re-optimize the physical plan at runtime using measured statistics, instead of relying only on compile-time estimates.

What AQE can do

  1. 1 shuffle (merge tiny partitions after shuffle)
  2. 2Switch join strategy (e.g., sort-merge -> broadcast when one side is small)
  3. 3Optimize skew joins (split skewed partitions)
Output
Build initial plan
   -> run shuffle / gather stats
   -> adapt: coalesce / change join / skew handling
   -> continue execution

Enable (Spark 3+)

Python
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

Why that matters

Static spark.sql.shuffle.partitions=200 is often wrong for both tiny and huge jobs. AQE reduces small-partition overhead and can fix some skew automatically.

Interview tip

AQE is not magic: bad data models, huge , and unnecessary still dominate. Use AQE plus good file layout and join design.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill