PySpark data engineering interview problem. Difficulty: advanced. Pattern: Medallion. About 22 minutes. Part of the Pro drill bank.
Merge a batch of updates into a target: updated keys replaced, new keys inserted, the rest untouched. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
target holds the current customers. batch holds incoming rows; the same customer_id can appear more than once in the batch with different updated_at values, and only the latest version of each key (by updated_at) counts. Return the target after applying the batch: keys in the batch take the latest batch row (the whole row) keys only in the batch are added keys only in the target are unchanged There is no Delta MERGE here, so build the result with ordinary DataFrame operations. Columns: customer_id, name, plan, updated_at. Assign the DataFrame to result.
Input: target customer_id | name | plan | updated_at 1 | Ann | basic | 2024-01-01 2 | Bo | pro | 2024-01-01 3 | Cy | basic | 2024-01-01 batch customer_id | name | plan | updated_at 2 | Bo | enterprise | 2024-02-01 4 | Di | basic | 2024-02-01 4 | Di | pro | 2024-02-05 3 | Cy | basic | 2024-02-01 Output: customer_id | name | plan | updated_at 1 | Ann | basic | 2024-01-01 2 | Bo | enterprise | 2024-02-01 3 | Cy | basic | 2024-02-01 4 | Di | pro | 2024-02-05 Customer 1 is untouched, 2 becomes enterprise, 3 is replaced by its batch row (new updated_at), and 4 is inserted with its latest version (pro).
Topics: lakebench, pyspark, upsert, left_anti, row_number, incremental.
More PySpark interview questions · All interview problems · Learn data engineering
Interview-style drill: Merge a batch of updates into a target: updated keys replaced, new keys inserted, the rest untouched.
`target` holds the current customers. `batch` holds incoming rows; the same `customer_id` can appear more than once in the batch with different `updated_at` values, and only the **latest** version of each key (by `updated_at`) counts. Return the target after applying the batch: - keys in the batch take the latest batch row (the whole row) - keys only in the batch are added - keys only in the target are unchanged There is no Delta MERGE here, so build the result with ordinary DataFrame operations. Columns: `customer_id`, `name`, `plan`, `updated_at`. Assign the DataFrame to `result`.