04. Debugging with Logs
Debugging with Logs in dbt
Debugging with logs is essential to identify failures, performance bottlenecks, and unexpected behavior in dbt models.
Proper log analysis ensures efficient troubleshooting and improves pipeline reliability.
Key Log Types
dbt run logs – Track model execution and SQL statements
dbt test logs – Track results of tests (uniqueness, not null, relationships)
Snapshot logs – Track historical versioning and changes
CLI debug mode – Provides detailed execution information
1. Enabling Debug Mode
- Use
--debugto see detailed logs during dbt runs:
dbt run --profiles-dir ./profiles --target prod --debug
- Provides information on:
- SQL compilation
- Model execution time
- Errors and stack traces
2. Reviewing dbt Logs
- Logs are stored in
logs/dbt.logby default - Contains:
- Timestamps for execution steps
- SQL queries executed
- Error messages and warnings
tail -f logs/dbt.log
- Use grep or search tools to filter relevant logs
3. Debugging Common Issues
Common scenarios:
- Compilation errors – Check
dbt compileoutput for SQL issues - Test failures – Review
dbt testlogs for failing assertions - Performance bottlenecks – Identify long-running models in logs
- Snapshot issues – Ensure correct configuration of
unique_keyandstrategy
4. Best Practices for Log Debugging
Enable --debug for troubleshooting complex issues
Review dbt logs after every production run
Centralize logs if running in orchestration tools (Airflow, Prefect)
Monitor execution times and identify slow models
Combine logging with notifications for proactive alerts