Tumbling-window counts with a watermark
Python data engineering interview problem. Difficulty: advanced. Pattern: Streaming. About 18 minutes. Part of the Pro drill bank.
Count streaming events in fixed time windows with watermark tracking and late-arrival dropping. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
Implement tumbling_window_counts(events: list[tuple[int, str]], window_size: int, allowed_lateness: int) -> list[dict] to aggregate an out-of-order event stream. Each event is a tuple (timestamp, event_id) arriving in streaming sequence. Windows are contiguous half-open intervals [W, W + window_size) where W = (timestamp // window_size) * window_size. Track the watermark: watermark = max_timestamp_seen - allowed_lateness. When an event arrives: If its timestamp < watermark, the event is considered late data and dropped. Otherwise, increment the count for its window. Update max_timestamp_seen and the watermark. Any window whose window_end <= watermark is finalized and emitted. At the end of the input stream, flush and emit any remaining unclosed windows that contain at least one accepted event. Return a list of records {"window_start": int, "window_end": int, "count": int} sorted by window_start.
Requirements
- Drop events with timestamp strictly less than the current watermark.
- Sort output records by window_start.
Constraints
- 0 <= len(events) <= 10,000
- window_size >= 1
- allowed_lateness >= 0
Examples
Input: tumbling_window_counts([(5, 'e1'), (10, 'e2'), (8, 'e3'), (25, 'e4'), (2, 'e_late')], 10, 5) Output: [{'window_start': 0, 'window_end': 10, 'count': 2}, {'window_start': 10, 'window_end': 20, 'count': 1}, {'window_start': 20, 'window_end': 30, 'count': 1}] Event e_late at timestamp 2 arrives after the watermark has reached 20 (25 - 5), so it is dropped as late data.
Topics: lakebench, python, watermark, out-of-order.
More Python interview questions · All interview problems · Learn data engineering
Tumbling-window counts with a watermark
Interview-style drill: Count streaming events in fixed time windows with watermark tracking and late-arrival dropping.
Implement `tumbling_window_counts(events: list[tuple[int, str]], window_size: int, allowed_lateness: int) -> list[dict]` to aggregate an out-of-order event stream. Each event is a tuple `(timestamp, event_id)` arriving in streaming sequence. - Windows are contiguous half-open intervals `[W, W + window_size)` where `W = (timestamp // window_size) * window_size`. - Track the watermark: `watermark = max_timestamp_seen - allowed_lateness`. - When an event arrives: - If its `timestamp < watermark`, the event is considered late data and dropped. - Otherwise, increment the count for its window. - Update `max_timestamp_seen` and the watermark. - Any window whose `window_end <= watermark` is finalized and emitted. - At the end of the input stream, flush and emit any remaining unclosed windows that contain at least one accepted event. - Return a list of records `{"window_start": int, "window_end": int, "count": int}` sorted by `window_start`.