0
Accepted answer
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
Team is split on approach A vs B. Looking for experiences from similar scale.
Context: What metrics do you alert on for batch pipelines?
Happy to share schema snippets or metrics if useful.
Accepted answer
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
Our dimension is slowly changing, does that change the join order?
Start with the execution plan, numbers beat guesses.
Sign in to reply.