Windowed Aggregation on Stream

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Streaming. About 20 minutes. Part of the Pro drill bank.

Streaming jobs often count events in fixed time windows, such as every 5 minutes. This is called a tumbling window. Each event falls into exactly one window, and each window is named by its start time.

The table events has event_id and event_time, all between 10:00 and 10:15 on 2024-01-01. Put each event into a 5-minute window and name the window by its start time: 10:00, 10:05 or 10:10. Count the events in each window as event_count. Return window_start and event_count, sorted by window_start. In this editor you can build window_start with a when chain on event_time. The editor creates the table events and ends with result = events. Change the last line to build your answer.

Requirements

• Assign each event to a window starting at 10:00, 10:05 or 10:10.
• Count the events in each window as event_count.
• Return window_start and event_count, sorted by window_start.
• Store the DataFrame in result.

Constraints

• A window is 5 minutes long: from its start up to, but not including, the next start.
• The window start is written as a timestamp text, such as 2024-01-01 10:00:00.
• The sample has 6 events and 3 windows.

Examples

Input: event times Output: window_start | event_count 2024-01-01 10:00:00 | 2 2024-01-01 10:05:00 | 2 2024-01-01 10:10:00 | 2 2 events land in each 5-minute bucket.

Topics: lakebench, pyspark, window, groupBy.

More PySpark interview questions · All interview problems · Learn data engineering