Dynamic task mapping vs many parallel operators

We saw the same issue, fixing the partition filter dropped runtime 60%.

In simple terms: your job is asking for way more data than it actually needs, and that extra data has to travel across the network or between machines, which is slow. Ask for less, or ask smarter.

Start with the execution plan, numbers beat guesses.

Start with the execution plan, numbers beat guesses.

+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.

What batch size or interval worked for you at similar scale?

Any downside to this approach with incremental models?

Another path: push the compute to the warehouse if the data's already there.

Worth measuring the serialized size before choosing broadcast.

This matches our runbook for skewed keys.