dbt · Incremental Models & Performance

Late-arriving data and lookback

Hardlate-arriving datalookbackwatermarkincremental

Interview question

How do you handle late-arriving data in incremental dbt models?

How to answer

The short answer

= rows whose event/update time is older than your last watermark, so a naive where updated_at > max(updated_at) misses them forever.

Lookback window pattern

Reprocess a trailing period every run:

SQL
{% if is_incremental() %}
where updated_at >= (
    select dateadd(day, -3, max(updated_at)) from {{ this }}
)
{% endif %}

Combined with merge (or overwrite for those days), late keys in the last 3 days get corrected.

Partition lookback (insert_overwrite style)

Output
each run: rebuild partitions for today-7 .. today
overwrite those partitions
older history stays untouched

Other tactics

  • Use a reliable updated_at from , not only created_at
  • Full refresh periodically for critical tables
  • Separate "corrections" stream merged by key
  • Increase lookback after incidents (temporary)

Trade-off

Interview tip: "Watermarks alone are not enough when data can arrive late. I use a lookback plus merge/overwrite, sized to the business's late-arrival ."