PySpark data engineering interview problem. Difficulty: advanced. Pattern: Skew. About 20 minutes. Part of the Pro drill bank.
Salt a skewed join key then join and drop the salt. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
Salt left with abs(hash(payload)) % 2, explode salt 0..1 on right, join on (user_id, salt), then return user_id, payload, name ordered by user_id, payload. Assign result.
Input: skewed left Output: user_id | payload | name 1 | a | Ada 1 | b | Ada 1 | c | Ada 1 | d | Ada 2 | e | Alan 3 | f | Grace Salting spreads the hot user_id=1 key.
Topics: lakebench, pyspark, salt, join.
More PySpark interview questions · All interview problems · Learn data engineering
Interview-style drill: Salt a skewed join key then join and drop the salt.
Salt `left` with `abs(hash(payload)) % 2`, explode salt 0..1 on `right`, join on (`user_id`, `salt`), then return `user_id`, `payload`, `name` ordered by `user_id`, `payload`. Assign `result`.