Skip to main content

03. Scheduling Jobs

Scheduling dbt Jobs​

Scheduling dbt jobs ensures that your models, snapshots, and tests run automatically and consistently in production.
Proper scheduling improves data reliability and aligns with business needs.


Key Scheduling Options​

  1. dbt Cloud Scheduler – Built-in scheduling for dbt Cloud users
  2. Apache Airflow / Prefect / Dagster – Orchestrate complex workflows
  3. Cron Jobs / Server Scheduler – Lightweight, simple automation

1. Using dbt Cloud Scheduler​

  • Navigate to dbt Cloud → Jobs → Create Job
  • Configure:
    • Environment: dev, staging, prod
    • Schedule: hourly, daily, weekly
    • Commands: dbt run, dbt test, dbt snapshot
  • Enable email/Slack notifications for success or failure

2. Using Airflow for Scheduling​

  • Define DAG with schedule_interval:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
'dbt_daily_pipeline',
start_date=datetime(2025, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:

dbt_run = BashOperator(
task_id='dbt_run',
bash_command='dbt run --profiles-dir /path/to/profiles --target prod'
)

dbt_test = BashOperator(
task_id='dbt_test',
bash_command='dbt test --profiles-dir /path/to/profiles --target prod'
)

dbt_run >> dbt_test
  • @daily → Runs the DAG once every day
  • Airflow also supports @hourly, cron expressions, or custom schedules

3. Using Cron Jobs​

  • Schedule via OS-level cron (Linux/Mac):
0 2 * * * cd /path/to/dbt_project && dbt run --target prod >> dbt_run.log 2>&1
  • Runs daily at 2 AM
  • Logs output to dbt_run.log
  • Lightweight alternative to Airflow for small projects

Best Practices​

Schedule based on data availability
Avoid overlapping runs; use Airflow or dbt Cloud concurrency settings
Monitor logs and notifications for failures
Include tests and snapshots in scheduled runs for data quality
Use version control to deploy jobs consistently across environments