A join can lose rows without any error when the keys do not match the way you think. Spark follows SQL rules, and several of them are easy to forget. The cure is to always compare counts before and after.
NULL keys never match
NULL = NULL is not true in SQL, so rows with a NULL join key drop out of an inner join, and appear with NULLs in an outer join. If NULLs should match, use the null-safe operator:
a.join(b, a.key.eqNullSafe(b.key)) # same as a.key <=> b.key in SQL
Be sure that is what you want. Matching all NULLs to each other creates a many-to-many join and can multiply rows badly.
Type mismatches
If one side is a string and the other an integer, Spark casts one to match. A string like "00123" becomes 123, so the leading zeros are lost, and different codes ("0123" and "123") can collide. A value like "N/A" cast to integer becomes NULL under the old default behaviour and then never matches. In Spark 4 with ANSI mode on, the same cast may raise an error, which is at least visible. Cast both sides explicitly to the same type on purpose.
Dirty strings
"IN-001 " with a trailing space does not equal "IN-001". Case differences also break equality. Trim and normalise the key before joining, ideally in the cleaning layer, not in each join.
Duplicate column names
After a.join(b, a.id == b.id), the result has two columns called id. Selecting id fails as ambiguous, and writing to Parquet may fail or silently keep only one. Join on the column name (a.join(b, "id")), which keeps one copy, or alias the columns first.
A quick reconciliation habit
print(left.count(), right.count(), joined.count()) left.join(right, "id", "left_anti").count() # rows that found no match
The anti join count tells you how many left rows found no partner. If it is higher than you expect, look at their key values. Also check for fan-out: if joined.count() is higher than the left count, the right side had duplicate keys.