Skip to main content

Defer & State Modified (Slim CI)

Imagine an Indian logistics company with 1,500 dbt models running on Snowflake.

If a Data Engineer in Bengaluru fixes a typo in a single downstream dashboard model (dim_delivery_metrics.sql), how do they test it?

  1. They could run dbt run, which builds all 1,500 models. This takes 45 minutes and burns $50 of Snowflake compute.
  2. They could run dbt run -s dim_delivery_metrics. This builds the model, but it looks for the upstream parent models (like stg_deliveries) in the developer's personal schema. Since the developer hasn't built stg_deliveries recently, the query fails with "Table not found."

To solve this, dbt introduced State Comparison and Deferral. This combination is often called Slim CI (Continuous Integration).

Topics Covered​

  • Understanding dbt State (manifest.json)
  • The state:modified selector
  • The --defer flag
  • Configuring Slim CI in dbt Cloud

1. Understanding dbt State (manifest.json)​

Every time dbt compiles a project, it generates a file called manifest.json in the target/ directory. This file is the "State". It contains the exact definition, SQL code, and schema of every single model at that moment in time.

If you have a production run that finished an hour ago, its manifest.json represents the current "Production State".

2. The state:modified Selector​

If you download the "Production State" (manifest.json) into your local environment, dbt can compare your local code against the production code.

By using the state:modified selector, dbt will identify exactly which files you have changed, and it will only build those models.

# Tell dbt where to find the Production manifest.json
dbt run --select state:modified --state ./prod-run-artifacts/

If you only changed dim_delivery_metrics.sql, this command will build exactly 1 model instead of 1,500.

3. The --defer Flag​

The command above still has a problem. When dim_delivery_metrics.sql executes, it uses {{ ref('stg_deliveries') }}. Because you only ran 1 model, dbt is looking for stg_deliveries in your personal dev schema (DEV_ARJUN), which doesn't exist.

This is where Deferral comes in.

When you add the --defer flag, you tell dbt: "If a parent model hasn't been modified, do not fail. Instead, 'defer' (point) the reference to the official production schema."

# The magic command for Slim CI
dbt run --select state:modified+ --state ./prod-run-artifacts/ --defer

(Notice the + sign. This tells dbt to run the modified model AND anything downstream of it that might be broken by the change).

What happens?

  1. dbt sees that dim_delivery_metrics is modified. It compiles it to build in DEV_ARJUN.
  2. dbt sees the {{ ref('stg_deliveries') }}.
  3. dbt realizes stg_deliveries was NOT modified.
  4. Instead of looking in DEV_ARJUN, dbt dynamically rewrites the compiled SQL to query the production table: FROM BEEMS_PROD.STAGING.stg_deliveries.

The query succeeds, using the actual production data as the upstream source, taking 5 seconds and burning $0.01 in Snowflake credits!

4. Configuring Slim CI in dbt Cloud​

Running this locally requires manually downloading the manifest.json file. In dbt Cloud, configuring Slim CI is fully automated.

When setting up a Continuous Integration (CI) job in dbt Cloud (a job triggered by a Pull Request):

  1. Create a new Job.
  2. Check the box for "Run on Pull Requests".
  3. Under Deferral, select the official Production Job as the "Environment to defer to".
  4. In the Commands section, replace the standard dbt run with the Slim CI command:
    dbt build --select state:modified+

(Note: dbt Cloud automatically injects the --defer and --state flags when you configure the Deferral setting in the UI, so you don't type them in the command box).

Now, when a developer opens a Pull Request on GitHub, dbt Cloud will automatically fetch the latest production manifest, find the modified files, and defer upstream queries to production. The PR testing finishes in 2 minutes instead of 45 minutes.

Summary​

  • manifest.json defines the "State" of a dbt project.
  • state:modified tells dbt to only run models whose code has changed compared to the state file.
  • --defer tells dbt to query the official production tables for any upstream parent models that weren't modified.
  • Combining these features creates Slim CI, which drastically reduces Snowflake compute costs and PR testing times.