Watermark for Late Data

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Late Data. About 20 minutes. Part of the Pro drill bank.

In streaming, events can arrive late. A watermark is a rule that says how late is too late. Spark remembers the biggest event time it has seen, and ignores events older than that time minus a delay. Without a watermark, Spark would have to keep state for old windows forever.

The table events has event_id and event_time. Treat the biggest event_time, 2024-01-01 10:25:00, as the latest event seen. Use a watermark of 30 minutes, so the cutoff is 10:25:00 minus 30 minutes, which is 09:55:00. Keep the events with event_time at or after 2024-01-01 09:55:00 and drop the older ones. Return event_id and event_time, sorted by event_time. The editor creates the table events and ends with result = events. Change the last line to build your answer.

Requirements

• Use the cutoff 2024-01-01 09:55:00.
• Keep events with event_time >= cutoff.
• Return event_id and event_time, sorted by event_time.
• Store the DataFrame in result.

Constraints

• The watermark is the maximum event time minus 30 minutes.
• An event exactly at the cutoff is kept (>=).
• The sample has 5 events, and 3 of them are on time.

Examples

Input: event times Output: event_id | event_time 1 | 2024-01-01 10:00:00 2 | 2024-01-01 10:10:00 4 | 2024-01-01 10:25:00 09:20 and 08:00 fall behind the watermark.

Topics: lakebench, pyspark, watermark, filter.

More PySpark interview questions · All interview problems · Learn data engineering