Read CSV with Explicit Schema

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Schema Drift. About 12 minutes. Free to practice.

Build a typed events frame from string columns without StructType CSV reads. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

When you read a CSV file, every value arrives as text, even the numbers. Before you can add up amounts or sort by an id, you have to turn those text values into real numbers. This is called casting. The editor gives you a small table called raw where every column is text, like a CSV that has just been read. Make a new table in which user_id is an integer and amount is a decimal number (a double). Keep event_type and ts as text. Sort by user_id. Save your final table in a variable called result. Do not call .collect() or .toPandas(), because those pull the data out of Spark into plain Python.

Requirements

• user_id must be an integer column.
• amount must be a double (decimal) column.
• event_type and ts stay as text.
• Sort the rows by user_id.
• Store the final DataFrame in result.

Constraints

• The input is the DataFrame raw, where every column is a string. It has at most 6 rows.
• Use .cast(...) on a column to change its type, for example F.col('user_id').cast('int').
• Do not call .collect() or .toPandas().

Examples

Input: raw rows with string user_id/amount Output: user_id | event_type | amount | ts 1 | purchase | 12.5 | 2024-01-01 10:00:00 2 | view | 3.0 | 2024-01-01 11:00:00 3 | purchase | 99.9 | 2024-01-02 09:00:00 4 | refund | 15.0 | 2024-01-02 10:00:00 Cast restores numeric types while preserving event_type and ts.

Topics: lakebench, pyspark, cast, schema.

More PySpark interview questions · All interview problems · Learn data engineering