Skip to main content

01. Running DBT in Production

Running dbt in Production​

Running dbt in production ensures that your transformations and analytics workflows are executed reliably and on schedule.
Production workflows typically involve automated scheduling, environment management, and monitoring.


Key Considerations​

Environment separation – dev, test, prod
Scheduling & Orchestration – ensure models run in the correct order
Logging & Monitoring – track success, failures, and performance
Version control & CI/CD – automate deployments using Git and CI tools


1. Environment Management​

Best practices:

  • Use separate Snowflake/Redshift/BigQuery schemas for dev, staging, and production
  • Define environment variables in profiles.yml for credentials and targets
  • Example profiles.yml snippet:
my_project:
target: prod
outputs:
dev:
type: snowflake
account: <account>
user: <dev_user>
password: <dev_password>
role: <dev_role>
database: my_db
warehouse: my_wh
schema: dev_schema
prod:
type: snowflake
account: <account>
user: <prod_user>
password: <prod_password>
role: <prod_role>
database: my_db
warehouse: my_wh
schema: prod_schema

2. Scheduling and Orchestration​

Options for running dbt in production:

  • Airflow / Prefect / Dagster → Orchestrate DAGs of dbt runs
  • Cron jobs / Scheduler → Simple periodic execution
  • dbt Cloud → Built-in scheduling and monitoring

Example Airflow DAG:

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG('dbt_prod_dag', start_date=datetime(2025, 1, 1), schedule_interval='@daily') as dag:
dbt_run = BashOperator(
task_id='dbt_run',
bash_command='dbt run --profiles-dir /path/to/profiles --target prod'
)

3. Logging and Monitoring​

  • Monitor dbt logs: /logs/dbt.log
  • Track success/failure status via orchestration tool alerts
  • Optional: Use Slack/email notifications for failures

Example dbt CLI notification with Bash:

if ! dbt run --target prod; then
echo "dbt run failed" | mail -s "DBT Failure Alert" team@example.com
fi

4. Version Control & CI/CD​

  • Store dbt project in Git
  • Use CI/CD pipelines to automatically deploy changes to production
  • Example flow:
    1. Developer pushes code to Git
    2. CI pipeline runs dbt test on dev/staging
    3. On success, pipeline runs dbt run --target prod

Best Practices​

Separate dev, staging, and prod environments
Schedule dbt runs via orchestration tools
Enable logging, monitoring, and alerts
Implement CI/CD pipelines for automated, reliable deployments
Test transformations in staging before deploying to production