Microbatch incremental strategy

The microbatch incremental strategy introduced in dbt 1.9 processes high-volume time-series data in discrete, deterministic time intervals rather than executing one monolithic query. You write SQL for a single time slice, and dbt automatically injects time-range filters for each batch based on an event timestamp. Each batch runs and can be retried independently, making large incremental pipelines resilient to failures.

How microbatch execution works

In traditional incremental strategies, dbt executes a single large query scanning everything since max updated_at. If the job fails mid-run, or if late-arriving records require reprocessing three days of data, the entire merge query must restart from scratch.

Microbatch changes this by breaking the run into bounded time intervals based on batch_size:

• The developer specifies event_time, batch_size (such as day or hour), and a start date with begin.
• dbt generates and runs an independent query for each interval.
• Batches can execute concurrently across available warehouse threads, significantly reducing total pipeline runtimes.

This interval-based design isolates failures to individual slices.

Model configuration

To configure a microbatch model, define the incremental strategy and event parameters in the model config block:

{{ config(
    materialized='incremental',
    incremental_strategy='microbatch',
    event_time='event_timestamp',
    batch_size='day',
    begin='2025-01-01',
    lookback=2
) }}

select
    event_id,
    user_id,
    event_name,
    event_timestamp
from {{ ref('stg_clickstream_events') }}

For filtering to push down efficiently, upstream models must also configure their own event_time in YAML. When configured, dbt automatically injects date filters into the upstream ref query during compilation.

Targeted reprocessing and backfills

The primary operational advantage of microbatch is targeted backfills. When you need to reprocess a specific week due to upstream source corrections, you pass explicit time boundaries on the command line:

dbt build --select fct_clickstream \
  --event-time-start "2026-03-01" \
  --event-time-end "2026-03-07"

dbt runs only the seven daily batches spanning that window. If the fourth day encounters an issue, you can retry that single batch without recomputing the entire table.