Testing Sources
In previous modules, we discussed how to test the models that we build in dbt (e.g., placing tests on fct_orders).
However, the most critical place to apply data quality tests is at the very beginning of the pipeline: The Sources.
Raw data is unpredictable. A third-party API might suddenly change its schema, or an ingestion tool like Fivetran might duplicate records. If you do not test the raw data, corrupt data will silently flow through your entire DAG and ruin your executive dashboards.
Topics Covered
- Why test sources?
- Configuring tests in
sources.yml - The
dbt buildworkflow - Source Freshness vs Source Testing
1. Why Test Sources?
Imagine your Indian logistics startup receives real-time delivery events via a Kafka stream into Snowflake.
Your stg_deliveries model assumes that the delivery_id is always unique and the weight_kg is never negative.
If a bug in the Kafka producer inserts duplicate delivery_ids, and you only test your final fct_deliveries table at the end of the DAG, you have a massive problem. The corrupt data has already been processed!
You must catch the bad data before it enters your transformation pipeline.
2. Configuring Tests in sources.yml
Applying tests to sources uses the exact same syntax as applying tests to models. You simply add the tests block under the column definitions in your sources.yml file.
# models/staging/sources.yml
version: 2
sources:
- name: logistics_api
database: BEEMS_PROD
schema: RAW
tables:
- name: delivery_events
description: "Raw delivery tracking events from Kafka."
columns:
- name: delivery_id
description: "Primary key of the delivery event."
tests:
- unique
- not_null
- name: weight_kg
description: "Weight of the package."
tests:
- not_null
# You can apply custom generic tests here too!
- is_positive
- name: delivery_status
tests:
- accepted_values:
values: ['DISPATCHED', 'IN_TRANSIT', 'DELIVERED', 'FAILED']
3. The dbt build Workflow
Now that you have defined tests on your source data, how do you execute them?
If you run dbt run, dbt will build your models. It will not run your tests.
If you run dbt test, dbt will test your sources, but it will test them at the same time as your models, which might be too late.
The correct command is dbt build.
When you run dbt build, dbt executes the DAG in strict topological order:
- It runs the tests on
source('logistics_api', 'delivery_events'). - If the source tests pass, it builds
stg_deliveries. - It runs the tests on
stg_deliveries. - If the staging tests pass, it builds
fct_deliveries.
If the source test fails (e.g., a duplicate delivery_id is found in the raw data), dbt instantly halts execution. It skips building stg_deliveries and fct_deliveries. This is called a Data Quality Gate, and it is the single most important concept in robust data engineering.
4. Source Freshness vs Source Testing
Do not confuse Source Testing with Source Freshness!
- Source Testing (
dbt test/dbt build): Executes SQL queries against the raw data to check the contents of the data (e.g., "Are there nulls? Are there duplicates?"). - Source Freshness (
dbt source freshness): Checks theMAX(loaded_at)timestamp of the table to see when the data was last updated. It does not care about the contents of the data, only its age.
A mature data pipeline implements both.
- First, check if the data arrived on time (
dbt source freshness). - Then, verify the data is not corrupt (
dbt build).
Summary
- Testing Sources is critical to prevent garbage data from entering your transformation pipeline.
- Define source tests exactly like model tests in your
sources.ymlfile. - Use
dbt buildto implement a Data Quality Gate. If a source test fails, downstream models will be skipped automatically. - Source Testing verifies data quality, while Source Freshness verifies data latency.