Deduplicate Stream

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Deduplication. About 20 minutes. Part of the Pro drill bank.

Streams often deliver the same event more than once, for example when a message is retried. Spark Structured Streaming has dropDuplicates to remove the repeats. The question for any dedup is always which copy to keep.

The table events has event_id, user_id and event_time. Some events appear twice. Remove the duplicates so that each event_id appears once, and keep the first copy (the one with the earlier event_time). Sort by event_id. The editor creates the table events and ends with result = events. Change the last line to build your answer. The simplest tool is dropDuplicates(["event_id"]). Be aware that on a real cluster Spark does not promise which copy it keeps, and to be sure you would use a window with row_number().

Requirements

• Keep 1 row for each event_id.
• Keep the first copy, with the earlier event_time.
• Return all 3 columns, sorted by event_id.
• Store the DataFrame in result.

Constraints

• The table has 5 rows with 3 different event_id values.
• The result has 3 rows.
• dropDuplicates takes a list of the columns that define a duplicate.

Examples

Input: duplicate event ids Output: event_id | user_id | event_time E1 | 1 | 2024-01-01 10:00:00 E2 | 2 | 2024-01-01 10:02:00 E3 | 3 | 2024-01-01 10:03:00 First occurrence per event_id wins.

Topics: lakebench, pyspark, dropDuplicates, streaming.

More PySpark interview questions · All interview problems · Learn data engineering