Combine sources with different schemas

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Schema Drift. About 14 minutes. Part of the Pro drill bank.

Union three frames whose columns differ, fill the missing ones with NULL and tag each row with its source. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Three feeds describe orders but have different columns: web: id, user, amount mobile: id, user, amount, platform store: id, user, store_id Combine all rows into one DataFrame with every column that appears in any feed (id, user, amount, platform, store_id) plus a column source set to 'web', 'mobile' or 'store'. Columns a feed does not have are NULL. No rows may be lost or duplicated. Assign the DataFrame to result.

Requirements

• One source column.
• Every column from every feed.

Constraints

• Column names are the same across feeds where they overlap.
• Row counts add up.

Examples

Input: web id | user | amount 1 | ann | 20 2 | bo | 35.5 mobile id | user | amount | platform 3 | cy | 12 | ios store id | user | store_id 4 | di | 7 Output: id | user | amount | source | platform | store_id 1 | ann | 20 | web | NULL | NULL 2 | bo | 35.5 | web | NULL | NULL 3 | cy | 12 | mobile | ios | NULL 4 | di | NULL | store | NULL | 7 Four rows in total. amount is NULL for the store row, platform is NULL for web and store rows, store_id is NULL for web and mobile rows.

Topics: lakebench, pyspark, unionByName, schema drift, lit.

More PySpark interview questions · All interview problems · Learn data engineering