27
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
DAG B needs to wait for DAG A to finish, but they're maintained by different teams and we'd rather not couple them through a shared metadata table. ExternalTaskSensor feels like the obvious tool but has some sharp edges around schedule alignment.
Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.
Simple way to think about it: caching is a bet that you'll ask the same question again soon. If you don't, you're just paying rent on memory for nothing.
Short version for anyone skimming: this is a classic case of the tool doing exactly what you told it to, not what you meant. Double check the assumption, not the code.
ExternalTaskSensor works well when both DAGs share the same schedule interval and logical date semantics. The sharp edge is usually execution_date mismatches when schedules differ, which needs an explicit execution_date_fn mapping.
Start with the execution plan, numbers beat guesses.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.