dbt Cloud Logs
When a dbt job fails in production at 3:00 AM, the first thing a Data Engineer does is look at the logs. dbt Cloud provides detailed, structured logs for every single run, allowing you to trace exactly which model failed, what SQL was executed, and what error the data warehouse returned.
Topics Covered
- Accessing dbt Cloud Logs
- Understanding the Run Interface
- Identifying Compilation vs Execution Errors
- Accessing the Compiled SQL
- Exporting Logs for External Monitoring
1. Accessing dbt Cloud Logs
To view logs in dbt Cloud:
- Navigate to the Deploy menu in the top navigation bar.
- Click on Runs.
- This page displays the history of all jobs executed in your dbt Cloud environment.
- Click on a specific Run ID (e.g.,
#1024) to view its details.
The Run History page provides high-level context: Was the run successful? How long did it take? Who (or what schedule) triggered it?
2. Understanding the Run Interface
Once you click into a specific Run, you will see a detailed, step-by-step breakdown. A typical dbt Cloud job contains multiple commands, for example:
dbt deps(Installs packages)dbt seed(Loads seed files)dbt run(Executes the models)dbt test(Executes data quality tests)
You can click on any of these individual steps to expand the logs for that specific command.
If the job failed, dbt Cloud will automatically highlight the specific step that failed with a red "X".
3. Identifying Compilation vs Execution Errors
When looking at the logs for a failed dbt run step, you must determine whether the failure was a Compilation Error or an Execution Error.
Compilation Errors
A compilation error means dbt could not translate your Jinja/SQL code into valid SQL. The code never reached the data warehouse.
- Example Log:
Compilation Error in model stg_customers (models/staging/stg_customers.sql): 'ref' is undefined. - Cause: You typed
{{ ref('customers') }}but forgot the curly braces, or referenced a model that doesn't exist. - Fix: Open the dbt Cloud IDE and fix the syntax.
Execution Errors
An execution error means dbt successfully compiled the SQL and sent it to the Data Warehouse (e.g., Snowflake), but the Data Warehouse rejected it.
- Example Log:
Database Error in model fct_orders (models/marts/fct_orders.sql): SQL compilation error: error line 1 at position 24 invalid identifier 'CUST_ID' - Cause: The column
CUST_IDdoes not exist in the underlying table, or you misspelled a column name. - Fix: Check the actual table schema in Snowflake to verify the column names.
4. Accessing the Compiled SQL
When an Execution Error occurs, the log snippet provided by the data warehouse is often not enough to solve the problem. You need to see the exact SQL query that dbt sent to the warehouse.
In the dbt Cloud Run Logs:
- Find the specific model that failed (e.g.,
fct_orders). - Click on the model name in the log output.
- Click the Details tab.
- You will see both the Source Code (with Jinja tags like
{{ ref('stg_orders') }}) and the Compiled Code (pure SQL with hardcoded table names likeBEEMS_PROD.STAGING.STG_ORDERS).
You can copy this Compiled SQL, paste it directly into your Snowflake or BigQuery console, and run it to debug exactly which line of SQL is causing the failure.
5. Structured Logging & Exporting
By default, dbt Cloud displays logs in a human-readable, console-friendly format.
However, modern DataOps teams often want to pipe these logs into external monitoring tools like Datadog, Splunk, or AWS CloudWatch.
dbt supports Structured Logging (JSON format). When structured logging is enabled (either via the dbt_project.yml or CLI flags), every log line is output as a JSON object containing the timestamp, log level (INFO, WARN, ERROR), and metadata about the specific node/model executing.
dbt Cloud allows you to stream these JSON logs to external systems via Webhooks, enabling you to build custom operational dashboards for your Indian data engineering team.
Summary
- dbt Cloud Logs provide a step-by-step history of every job execution.
- Compilation Errors mean dbt couldn't parse your code. Execution Errors mean the Data Warehouse rejected the query.
- Always check the Compiled Code to see the exact SQL that was sent to the database.
- Use Structured Logging to export run metadata to external monitoring tools.