Broadcast join silently falls back to sort-merge, how do you actually tell

Prefer a staging table plus validation gate before promoting to prod tables.

Careful with NULL in join keys, they'll drop rows in an inner join.

In our case the root cause was an implicit cast preventing pushdown.

Run df.join(...).explain() and look for BroadcastHashJoin vs SortMergeJoin in the physical plan. The hint is a request, not a guarantee, Spark can override it.

Prefer a staging table plus validation gate before promoting to prod tables.

Sometimes the real fix is a product change so you stop needing that join at all.

In our case the root cause was an implicit cast preventing pushdown.

Plain English: the system prefers to guess a good-enough plan than the perfect plan, because figuring out the perfect plan would take longer than just running the good-enough one.