Defer & State Modified (Slim CI)
Imagine an Indian logistics company with 1,500 dbt models running on Snowflake.
If a Data Engineer in Bengaluru fixes a typo in a single downstream dashboard model (dim_delivery_metrics.sql), how do they test it?
- They could run
dbt run, which builds all 1,500 models. This takes 45 minutes and burns $50 of Snowflake compute. - They could run
dbt run -s dim_delivery_metrics. This builds the model, but it looks for the upstream parent models (likestg_deliveries) in the developer's personal schema. Since the developer hasn't builtstg_deliveriesrecently, the query fails with "Table not found."
To solve this, dbt introduced State Comparison and Deferral. This combination is often called Slim CI (Continuous Integration).
Topics Covered
- Understanding dbt State (
manifest.json) - The
state:modifiedselector - The
--deferflag - Configuring Slim CI in dbt Cloud
1. Understanding dbt State (manifest.json)
Every time dbt compiles a project, it generates a file called manifest.json in the target/ directory. This file is the "State". It contains the exact definition, SQL code, and schema of every single model at that moment in time.
If you have a production run that finished an hour ago, its manifest.json represents the current "Production State".
2. The state:modified Selector
If you download the "Production State" (manifest.json) into your local environment, dbt can compare your local code against the production code.
By using the state:modified selector, dbt will identify exactly which files you have changed, and it will only build those models.
# Tell dbt where to find the Production manifest.json
dbt run --select state:modified --state ./prod-run-artifacts/
If you only changed dim_delivery_metrics.sql, this command will build exactly 1 model instead of 1,500.
3. The --defer Flag
The command above still has a problem. When dim_delivery_metrics.sql executes, it uses {{ ref('stg_deliveries') }}. Because you only ran 1 model, dbt is looking for stg_deliveries in your personal dev schema (DEV_ARJUN), which doesn't exist.
This is where Deferral comes in.
When you add the --defer flag, you tell dbt: "If a parent model hasn't been modified, do not fail. Instead, 'defer' (point) the reference to the official production schema."
# The magic command for Slim CI
dbt run --select state:modified+ --state ./prod-run-artifacts/ --defer
(Notice the + sign. This tells dbt to run the modified model AND anything downstream of it that might be broken by the change).
What happens?
- dbt sees that
dim_delivery_metricsis modified. It compiles it to build inDEV_ARJUN. - dbt sees the
{{ ref('stg_deliveries') }}. - dbt realizes
stg_deliverieswas NOT modified. - Instead of looking in
DEV_ARJUN, dbt dynamically rewrites the compiled SQL to query the production table:FROM BEEMS_PROD.STAGING.stg_deliveries.
The query succeeds, using the actual production data as the upstream source, taking 5 seconds and burning $0.01 in Snowflake credits!
4. Configuring Slim CI in dbt Cloud
Running this locally requires manually downloading the manifest.json file. In dbt Cloud, configuring Slim CI is fully automated.
When setting up a Continuous Integration (CI) job in dbt Cloud (a job triggered by a Pull Request):
- Create a new Job.
- Check the box for "Run on Pull Requests".
- Under Deferral, select the official Production Job as the "Environment to defer to".
- In the Commands section, replace the standard
dbt runwith the Slim CI command:dbt build --select state:modified+
(Note: dbt Cloud automatically injects the --defer and --state flags when you configure the Deferral setting in the UI, so you don't type them in the command box).
Now, when a developer opens a Pull Request on GitHub, dbt Cloud will automatically fetch the latest production manifest, find the modified files, and defer upstream queries to production. The PR testing finishes in 2 minutes instead of 45 minutes.
Summary
manifest.jsondefines the "State" of a dbt project.state:modifiedtells dbt to only run models whose code has changed compared to the state file.--defertells dbt to query the official production tables for any upstream parent models that weren't modified.- Combining these features creates Slim CI, which drastically reduces Snowflake compute costs and PR testing times.