How many files per partition is too many for Parquet?

We saw the same issue, fixing the partition filter dropped runtime 60%.

Tried that, partial improvement but lag still spikes on redeploy.

Check whether AQE is disabled in your Spark conf, skew join handling helped us a lot here.

This matches our runbook for skewed keys.

Plain English: the system prefers to guess a good-enough plan than the perfect plan, because figuring out the perfect plan would take longer than just running the good-enough one.

Note that merge on Delta still needs unique keys defined correctly.

Start with the execution plan, numbers beat guesses.