PySpark data engineering interview problem. Difficulty: advanced. Pattern: Schema Drift. About 22 minutes. Part of the Pro drill bank.
Promote every nested struct field to a parent_child column and expand arrays without hardcoding any column name. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
orders is read from JSON and has nested structs and an array of structs: customer (with a nested address) and items. Produce one flat DataFrame: every field of a struct becomes a top-level column named parent_child (nested structs follow the same rule: customer_address_city) every array becomes rows, one per element; an order whose array is empty is still returned, with NULL in the element columns the code must work from the DataFrame's own schema. Do not write the names customer, address or items as literals Sort by id. Assign the DataFrame to result.
Input: raw = spark.sparkContext.parallelize([ '{"id": 1, "customer": {"name": "Ann", "address": {"city": "Pune", "zip": "411001"}}, "items": [{"sku": "A", "qty": 2}, {"sku": "B", "qty": 1}]}', '{"id": 2, "customer": {"name": "Bo", "address": {"city": "Oslo", "zip": "0150"}}, "items": []}', '{"id": 3, "customer": {"name": "Cy", "address": {"city": "Rome", "zip": null}}, "items": [{"sku": "C", "qty": 5}]}', ]) orders = spark.read.json(raw) Output: id | customer_name | items_sku | items_qty | customer_address_city | customer_address_zip 1 | Ann | A | 2 | Pune | 411001 1 | Ann | B | 1 | Pune | 411001 2 | Bo | NULL | NULL | Oslo | 0150 3 | Cy | C | 5 | Rome | NULL Order 1 has two items so it produces two rows. Order 2 has an empty items array and is kept with NULL sku and qty. The nested address fields become customer_address_city and customer_address_zip.
Topics: lakebench, pyspark, json, struct, explode_outer, schema.
More PySpark interview questions · All interview problems · Learn data engineering
Interview-style drill: Promote every nested struct field to a parent_child column and expand arrays without hardcoding any column name.
`orders` is read from JSON and has nested structs and an array of structs: `customer` (with a nested `address`) and `items`. Produce one flat DataFrame: - every field of a struct becomes a top-level column named `parent_child` (nested structs follow the same rule: `customer_address_city`) - every array becomes rows, one per element; an order whose array is empty is still returned, with NULL in the element columns - the code must work from the DataFrame's own schema. Do not write the names `customer`, `address` or `items` as literals Sort by `id`. Assign the DataFrame to `result`.