Handle Schema Drift

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Schema Drift. About 16 minutes. Part of the Pro drill bank.

Data sources change over time. A team may add a column to a table, or remove one, without telling you. This is called schema drift, and a pipeline that does not check for it can break or lose data without any warning. A drift report compares the old column list with the new one.

You have two small tables, old and new, which hold the same data at two points in time. Compare their column names. Return 1 row with 3 columns: added_columns, the names that are in new but not in old (joined with commas); removed_columns, the names that are in old but not in new (joined with commas); and type_changes, which is the text none for this data. Use an empty text for a list with no names. The editor creates both tables and ends with result = old. Change the last line to build the 1-row answer, for example from the Python lists old.columns and new.columns.

Requirements

• Find the columns that exist only in new and join them with commas into added_columns.
• Find the columns that exist only in old and join them with commas into removed_columns.
• Set type_changes to the text none.
• Return exactly 1 row with the 3 columns, in result.

Constraints

• old has id and color. new has id, color and size.
• If there are no added or removed columns, use an empty text ('').
• .columns gives the list of column names of a DataFrame.

Examples

Input: old vs new Output: added_columns | removed_columns | type_changes size | | none size appears only on new.

Topics: lakebench, pyspark, schema.

More PySpark interview questions · All interview problems · Learn data engineering