Read from Kafka Stream

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Streaming. About 20 minutes. Part of the Pro drill bank.

When Spark reads from Kafka, each message arrives as one text value, often JSON. Before you can analyze it, you must turn this text into columns. get_json_object reads one field from a JSON string using a path such as $.user_id.

The table raw has 1 column, value, which holds a JSON string for each message, for example {"user_id":1,"event_type":"click","ts":"2024-01-01 10:00:00"}. Extract 3 columns with F.get_json_object: user_id (cast to an integer), event_type and ts. Sort by ts. The editor creates the table raw and ends with result = raw. Change the last line to build your answer.

Requirements

• Extract user_id with $.user_id and cast it to int.
• Extract event_type with $.event_type.
• Extract ts with $.ts.
• Sort by ts and store the DataFrame in result.

Constraints

• get_json_object always returns text, so the user id must be cast to int.
• The JSON paths start with $..
• The result has 4 rows and 3 columns.

Examples

Input: json values Output: user_id | event_type | ts 1 | click | 2024-01-01 10:00:00 2 | view | 2024-01-01 10:01:00 1 | purchase | 2024-01-01 10:02:00 3 | click | 2024-01-01 10:03:00 JSON fields become columns.

Topics: lakebench, pyspark, json, get_json_object.

More PySpark interview questions · All interview problems · Learn data engineering