The short answer
(AQE) lets Spark re-optimize the physical plan at runtime using measured statistics, instead of relying only on compile-time estimates.
What AQE can do
- 1 shuffle (merge tiny partitions after shuffle)
- 2Switch join strategy (e.g., sort-merge -> broadcast when one side is small)
- 3Optimize skew joins (split skewed partitions)
Build initial plan
-> run shuffle / gather stats
-> adapt: coalesce / change join / skew handling
-> continue executionEnable (Spark 3+)
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")Why that matters
Static spark.sql.shuffle.partitions=200 is often wrong for both tiny and huge jobs. AQE reduces small-partition overhead and can fix some skew automatically.
Interview tip
AQE is not magic: bad data models, huge , and unnecessary still dominate. Use AQE plus good file layout and join design.