The short answer
In advanced streaming pipelines, many analytical requirements cannot be expressed using simple stateless transformations. Use cases like user sessionization, detecting fraud sequences across disparate financial events, and deduplicating continuous clickstreams require maintaining historical state across time.
Apache Beam on Google Cloud Dataflow provides the State and Timer API for stateful stream processing.
Persistent state backends in Dataflow
When a DoFn declares state variables (such as ValueState, BagState, or MapState), Dataflow keys state strictly per key and per window.
Dataflow stores state in an optimized embedded local state backend backed by local persistent disk or SSD, rather than keeping all historical objects in JVM memory. This prevents out-of-memory errors when tracking state for millions of active keys.
public class DeduplicationDoFn extends DoFn<Event, Event> {
@StateId("seenTimestamp")
private final StateSpec<ValueState<Long>> seenSpec = StateSpecs.value();
@TimerId("expiryTimer")
private final TimerSpec timerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);
@ProcessElement
public void processElement(
ProcessContext context,
@StateId("seenTimestamp") ValueState<Long> seenState,
@TimerId("expiryTimer") Timer expiryTimer) {
// Evaluate state and set cleanup timer
}
}The operational trap: Uncollected expired state
The single most dangerous anti-pattern in stateful stream processing is failing to garbage-collect state.
If a streaming pipeline processes 50 million unique session keys daily and writes values into BagState without clearing them, the state storage volume grows indefinitely. Over weeks, worker disks fill up, operations stall, and pipeline worker autoscaling reaches maximum capacity.
To prevent memory and disk leaks, always pair state variables with an event-time timer. When the watermark advances past the session expiration window, the timer fires, clears the state (seenState.clear()), and frees physical disk blocks.