3
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
This worked in dev on sample data but fails at full volume. Details:
Context: EMR on EKS vs traditional EMR for Spark, ops overhead?
Happy to share schema snippets or metrics if useful.
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
Small nit: the broadcast hint gets ignored once the table is over threshold, check the UI to confirm.
Any downside to this approach with incremental models?
Sign in to reply.