Infer Schema vs Explicit Schema

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Schema Drift. About 16 minutes. Part of the Pro drill bank.

When you read a CSV file, Spark can either guess the column types (inferSchema) or you can tell it the types (an explicit schema). Guessing is convenient but can go wrong, and one bad value can change the guess for the whole column. A safer habit is to read everything as text and cast it yourself.

The table raw has two text columns, user_id and amount, as if they were just read from a CSV file. One of the amounts is not a number: x. Cast user_id to an integer and amount to a double. A value that cannot be cast becomes NULL. Keep only the rows where amount is not NULL, so the bad row is removed. Return the table sorted by user_id. The editor creates raw and ends with result = raw. Change the last line to build your answer.

Requirements

• Cast user_id to int and amount to double.
• Remove rows where amount is NULL after the cast.
• Keep both columns and sort by user_id.
• Store the DataFrame in result.

Constraints

• Use .cast("int") and .cast("double").
• A failed cast gives NULL. It does not raise an error.
• Filter after the cast, because the NULL only exists once the cast is done.

Examples

Input: string raw Output: user_id | amount 1 | 10.5 2 | 20.0 4 | 40.25 Row 3 casts to null and drops.

Topics: lakebench, pyspark, cast, schema.

More PySpark interview questions · All interview problems · Learn data engineering