Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Slim CI with state:modified and defer

dbt · Workflow, CI & Performance

Slim CI with state:modified and defer

Harddbt-48
slim-cistate-modifieddefercost-optimization

Question

How do you run only changed models in CI?

Solution

To run only changed models in continuous integration, you use dbt Slim CI by pairing state comparison with execution deferral. By passing state pointing to production artifacts and running dbt build with state:modified+ and defer, dbt builds only the modified models and their downstream dependents while reading unchanged upstream models directly from production.

State comparison mechanics

When dbt compiles a project, it writes a comprehensive graph description to target/manifest.json. Slim CI relies on comparing the manifest of your current pull request branch against a production manifest.json downloaded from cloud storage.

The selection flag state:modified+ inspects differences between the two manifests:

  • It identifies any model where the SQL, YAML configuration, or upstream dependencies changed.
  • The trailing plus sign instructs dbt to include all downstream children that depend on the modified models.
  • Any model untouched by the pull request is completely skipped.

This shrinks a 500-model pipeline run down to just the handful of models impacted by the pull request.

Combining modified selection and deferral

Selecting modified models alone is not enough. If you modify a mart table like fct_orders, its query still needs to read from upstream staging models like stg_orders. In an empty CI sandbox schema, those staging tables do not exist.

The defer flag resolves this challenge. When dbt compiles a reference to an unselected upstream model, defer redirects the query to read from the production database schema instead of the local CI schema:

dbt build \
  --select state:modified+ \
  --state path/to/prod/artifacts \
  --defer

The modified fct_orders is built inside the temporary CI schema, but it queries existing production data from prod.stg_orders. On warehouses that support zero-copy cloning like Snowflake and BigQuery, teams can also combine deferral with dbt clone to populate staging environments instantly. This saves substantial compute cost and runtime.

Operational requirements in CI pipelines

To make Slim CI work reliably, your automated CI workflow needs three setup steps:

  • Store production artifacts like manifest.json in object storage or fetch them via the dbt Cloud metadata API after each production deployment.
  • Download that production manifest.json into the CI container before executing test commands.
  • Run dbt build instead of separate run and test commands so each modified node and its tests execute together in topological order.

This workflow ensures pull requests complete testing in minutes while keeping cloud warehouse bills under control.

PreviousNext