Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Late-arriving data and lookback

dbt · Incremental Models & Performance

Late-arriving data and lookback

Harddbt-26
late-arriving datalookbackwatermarkincremental

Question

How do you handle late-arriving data in incremental dbt models?

Solution

Late-arriving data = rows whose event/update time is older than your last watermark, so a naive where updated_at > max(updated_at) misses them forever.

Lookback window pattern

Reprocess a trailing period every run:

{% if is_incremental() %}
where updated_at >= (
    select dateadd(day, -3, max(updated_at)) from {{ this }}
)
{% endif %}

Combined with merge (or partition overwrite for those days), late keys in the last 3 days get corrected.

Partition lookback (insert_overwrite style)

each run: rebuild partitions for today-7 .. today
overwrite those partitions
older history stays untouched

Other tactics

  • Use a reliable updated_at from CDC, not only created_at
  • Full refresh periodically for critical tables
  • Separate "corrections" stream merged by key
  • Increase lookback after incidents (temporary)

Trade-off

longer lookback → fewer missed late rows, more compute each run
shorter lookback → cheaper, higher miss risk

Interview tip: "Watermarks alone are not enough when data can arrive late. I use a lookback plus merge/overwrite, sized to the business's late-arrival SLA."

PreviousNext