Skip to main content

dbt Artifacts for Data Observability

Whenever you run a dbt command (dbt run, dbt test, dbt docs generate), dbt quietly produces a set of JSON files in the target/ directory. These files are called Artifacts.

While these files are used by dbt under the hood (e.g., to render the dbt Docs website or power Slim CI), they are also the absolute best source of metadata for Data Observability.

Topics Covered​

  • What is Data Observability?
  • The Three Core Artifacts
  • Using Artifacts for Custom Monitoring
  • Third-Party Observability Tools

1. What is Data Observability?​

Data Observability is the practice of monitoring the health, performance, and reliability of your data pipelines.

If you are the Lead Data Engineer at an Indian telecom company, you need to know:

  • Did the nightly pipeline fail?
  • Is the fct_daily_usage table taking 30 minutes longer to build than it did last week?
  • Did the not_null test on the customer_phone column suddenly fail today?

You can answer all of these questions programmatically by reading the dbt Artifacts.

2. The Three Core Artifacts​

dbt produces three primary JSON artifacts that contain everything you need to know about your project.

1. manifest.json​

This is the holy grail of your dbt project. It contains the exact definition of every single node in your DAG (models, tests, sources, seeds, exposures). It contains the raw SQL, the compiled SQL, the configured tags, and the exact upstream/downstream dependencies. It represents the state of your project.

2. run_results.json​

Produced after a dbt run or dbt test. It contains the exact execution details of that specific invocation. It tells you:

  • Which models succeeded, failed, or were skipped.
  • Exactly how many seconds it took to build each model.
  • How many rows failed a specific data test.

3. catalog.json​

Produced by dbt docs generate. It contains information pulled directly from the data warehouse's Information Schema, such as column data types and exact row counts for the physical tables in Snowflake.

3. Using Artifacts for Custom Monitoring​

Because these artifacts are just JSON files, you can upload them to your data warehouse and query them using SQL!

Many advanced data teams use the dbt-artifacts package. This package automatically takes your run_results.json and loads it into a Snowflake table.

Once the data is in Snowflake, you can build a Tableau dashboard on top of it. (Imagine a dashboard showing the exact execution time of every dbt model over the last 90 days. If the line graph spikes, you immediately know a model needs performance tuning!)

4. Third-Party Observability Tools​

Reading JSON files manually is tedious. Thankfully, an entire industry of "Data Observability Platforms" has emerged that integrate directly with dbt Artifacts.

Tools like Monte Carlo, Datafold, and Metaplane connect to your dbt Cloud account or GitHub repository.

  1. They ingest your manifest.json and run_results.json automatically.
  2. They use machine learning to understand the historical runtimes and row counts of your models.
  3. If stg_payments usually takes 2 minutes to build, and today it takes 15 minutes, Monte Carlo will instantly ping your Slack channel with an Anomaly Alert.
  4. Because they have the manifest.json, they will trace the lineage and tell you exactly which downstream Executive Dashboards are currently delayed due to the performance bottleneck.

Summary​

  • Data Observability is the monitoring of pipeline health, freshness, and performance.
  • dbt generates JSON Artifacts (manifest.json, run_results.json, catalog.json) in the target/ folder after every run.
  • You can parse these artifacts yourself (using tools like the dbt-artifacts package) to build custom monitoring dashboards.
  • Third-party observability platforms ingest these artifacts to provide automated anomaly detection and impact analysis alerting.