Synapse dedicated vs serverless for ad-hoc analytics

Prefer a staging table plus validation gate before promoting to prod tables.

+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.

Agree on the staging table swap. Atomic promote prevented partial reads.

Start with the execution plan, numbers beat guesses.

ELI5: think of it like a phone book. If it's sorted by last name and you search by last name, that's fast. Search by first name instead and you're flipping through every page.

In our case the root cause was an implicit cast preventing pushdown.

Idempotent writes with merge keys saved us during backfills.