Add Missing Columns

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Schema Drift. About 16 minutes. Part of the Pro drill bank.

A downstream table expects the columns user_id, email and phone. But the source has stopped sending phone. If you leave the column out, everything that reads the table breaks. The usual fix is to add the missing column with empty values (NULL), so the shape stays the same.

The table df has user_id and email, but the output must also have a phone column. Add phone as a column that is empty (NULL) for every row. Return user_id, email and phone in this order, sorted by user_id. The editor creates the table df and ends with result = df. Change the last line to build your answer. Use F.lit(None) to create a NULL value.

Requirements

• Add a phone column with NULL in every row.
• Return the columns in this order: user_id, email, phone.
• Sort by user_id.
• Store the DataFrame in result.

Constraints

• Use F.lit(None) for the NULL value. You may cast it to string with .cast("string").
• All 3 rows must stay.
• The new column must come last, after email.

Examples

Input: users without phone Output: user_id | email | phone 1 | [email protected] | None 2 | [email protected] | None 3 | [email protected] | None Missing column added as null.

Topics: lakebench, pyspark, withColumn, lit.

More PySpark interview questions · All interview problems · Learn data engineering