Validate Data Types

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Data Quality. About 16 minutes. Part of the Pro drill bank.

A pipeline should check the types of its columns before it trusts them. A column that should be a number but arrives as text can break a sum or a join without any error. A simple validation step compares the expected type with the actual type, and reports only the mismatches.

In this exercise, user_id should be an integer and amount should be a double. Build a small report with the columns column_name, expected_type and actual_type, listing only the columns whose actual type does not match the expected one. If every column is fine, the report is empty (zero rows, but with the 3 columns). The editor creates the table df and a small helper table probe that lists each column and its type. Change the last line so result holds only the mismatches. For this table, both columns have the right type, so the expected answer is an empty table with the 3 columns.

Requirements

• Compare expected_type and actual_type for each checked column.
• Return only the columns where the two do not match.
• The result has the columns column_name, expected_type, actual_type.
• If there are no mismatches, return an empty DataFrame with these columns. Store it in result.

Constraints

• The checked columns are user_id (expected int) and amount (expected double).
• An empty result is correct when every type matches.
• You can read the real types of a DataFrame with df.dtypes.

Examples

Input: typed df Output: column_name | expected_type | actual_type No mismatches for the provided fixture.

Topics: lakebench, pyspark, schema, validation.

More PySpark interview questions · All interview problems · Learn data engineering