01. Running DBT in Production
Running dbt in Production
Running dbt in production ensures that your transformations and analytics workflows are executed reliably and on schedule.
Production workflows typically involve automated scheduling, environment management, and monitoring.
Key Considerations
Environment separation – dev, test, prod
Scheduling & Orchestration – ensure models run in the correct order
Logging & Monitoring – track success, failures, and performance
Version control & CI/CD – automate deployments using Git and CI tools
1. Environment Management
Best practices:
- Use separate Snowflake/Redshift/BigQuery schemas for dev, staging, and production
- Define environment variables in
profiles.ymlfor credentials and targets - Example
profiles.ymlsnippet:
my_project:
target: prod
outputs:
dev:
type: snowflake
account: <account>
user: <dev_user>
password: <dev_password>
role: <dev_role>
database: my_db
warehouse: my_wh
schema: dev_schema
prod:
type: snowflake
account: <account>
user: <prod_user>
password: <prod_password>
role: <prod_role>
database: my_db
warehouse: my_wh
schema: prod_schema
2. Scheduling and Orchestration
Options for running dbt in production:
- Airflow / Prefect / Dagster → Orchestrate DAGs of dbt runs
- Cron jobs / Scheduler → Simple periodic execution
- dbt Cloud → Built-in scheduling and monitoring
Example Airflow DAG:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG('dbt_prod_dag', start_date=datetime(2025, 1, 1), schedule_interval='@daily') as dag:
dbt_run = BashOperator(
task_id='dbt_run',
bash_command='dbt run --profiles-dir /path/to/profiles --target prod'
)
3. Logging and Monitoring
- Monitor dbt logs:
/logs/dbt.log - Track success/failure status via orchestration tool alerts
- Optional: Use Slack/email notifications for failures
Example dbt CLI notification with Bash:
if ! dbt run --target prod; then
echo "dbt run failed" | mail -s "DBT Failure Alert" team@example.com
fi
4. Version Control & CI/CD
- Store dbt project in Git
- Use CI/CD pipelines to automatically deploy changes to production
- Example flow:
- Developer pushes code to Git
- CI pipeline runs
dbt teston dev/staging - On success, pipeline runs
dbt run --target prod
Best Practices
Separate dev, staging, and prod environments
Schedule dbt runs via orchestration tools
Enable logging, monitoring, and alerts
Implement CI/CD pipelines for automated, reliable deployments
Test transformations in staging before deploying to production