Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing

Watermark for Late Data

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Late Data. About 20 minutes. Part of the Pro drill bank.

Drop events older than 30 minutes behind the max event_time. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Treat max event_time as 2024-01-01 10:25:00. Keep rows with event_time >= 2024-01-01 09:55:00 (30-minute watermark). Order by event_time. Assign result.

Constraints

  • 30-minute watermark behind 10:25.
  • Order by event_time.

Examples

Input: event times Output: event_id | event_time 1 | 2024-01-01 10:00:00 2 | 2024-01-01 10:10:00 4 | 2024-01-01 10:25:00 09:20 and 08:00 fall behind the watermark.

Topics: lakebench, pyspark, watermark, filter.

More PySpark interview questions · All interview problems · Learn data engineering

advanced

Watermark for Late Data

Interview-style drill: Drop events older than 30 minutes behind the max event_time.

Treat max event_time as 2024-01-01 10:25:00. Keep rows with `event_time >= 2024-01-01 09:55:00` (30-minute watermark). Order by `event_time`. Assign `result`.