Drop Duplicates

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Deduplication. About 12 minutes. Free to practice.

Keep the first row per user_id + event_type. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Events are sometimes recorded more than once, for example because of a retry. In this table df, the same user can have the same event type several times. You only want to keep 1 row for each pair of user and event type, the first one that appears. Remove duplicate rows based on the pair (user_id, event_type), keeping the first occurrence in the table. Sort the result by user_id and then event_type. Save your final table in result.

Requirements

• Return 1 row for each pair of user_id and event_type.
• Keep the first row of each pair, as the data is listed in the input.
• Keep all the columns (user_id, event_type, ts).
• Sort by user_id and then event_type.

Constraints

• The input table is df with the columns user_id, event_type and ts.
• Only user_id and event_type decide what counts as a duplicate. ts can differ.
• Use dropDuplicates(['user_id', 'event_type']).

Examples

Input: duplicate events Output: user_id | event_type | ts 1 | purchase | 2024-01-01 10:00:00 2 | view | 2024-01-01 09:00:00 3 | purchase | 2024-01-02 08:00:00 Later duplicate timestamps for the same key are discarded.

Topics: lakebench, pyspark, dropDuplicates.

More PySpark interview questions · All interview problems · Learn data engineering