Cloud · Google Cloud Data Platform

Cloud Dataflow windowing and late data handling with watermarks

gcpMediumgcpdataflowapache-beamwindowing

Interview question

How does Apache Beam on Cloud Dataflow handle out-of-order streaming events using watermarks and triggers?

How to answer

The short answer

In real-world streaming data platforms, events generated on mobile devices, IoT sensors, or remote servers frequently arrive out of chronological order due to intermittent cellular network disconnections or processing delays.

Apache Beam running on Google Cloud Dataflow provides a streaming execution model based on the MillWheel framework to process out-of-order events accurately.

Event time versus processing time

  • Event Time: The timestamp recorded when the event actually occurred on the client device (such as 14:02:15).
  • Processing Time: The timestamp when the Cloud Dataflow worker receives and processes the record (such as 14:08:30, over 6 minutes later).

Calculating metrics (like hourly click counts) based on processing time corrupts analytics. Dataflow groups and windows records strictly by event time.

Watermarks: Tracking time progression

A watermark is Dataflow's computational estimate of how far event time has progressed across all input streams.

When Dataflow's watermark passes 15:00:00, the engine assumes that all events generated prior to 3:00 PM have been ingested. The engine can now close the 2:00 PM to 3:00 PM window and emit final aggregation results.

Handling stragglers: Allowed Lateness and triggers

What happens when a mobile user reconnects to the network hours later, transmitting an event timestamped 14:45:00 after the 3:00 PM window already fired?

Beam provides two controls:

  • Allowed Lateness: Specifies how long the pipeline keeps window state active after the watermark passes.
  • Triggers and Accumulation: Define when window updates are emitted.
JAVA
PCollection<Event> windowedEvents = events.apply(
    Window.<Event>into(FixedWindows.of(Duration.standardHours(1)))
          .triggering(AfterWatermark.pastEndOfWindow()
              .withLateFirings(AfterPane.elementCountAtLeast(1)))
          .withAllowedLateness(Duration.standardDays(1))
          .accumulatingFiredPanes());

When late data arrives within the allowed lateness window, Dataflow emits an updated refinement pane to downstream sinks without dropping valid business events.