Skip to main content

02. Orchestrating with Airflow

Orchestrating dbt with Airflow​

Apache Airflow is a popular orchestration tool for scheduling and managing dbt workflows in production.
It allows you to define Directed Acyclic Graphs (DAGs) for automated execution of dbt tasks.


Why Use Airflow for dbt?​

Schedule dbt runs with flexible intervals
Manage dependencies between models
Integrate notifications and monitoring
Combine dbt tasks with other ETL/ELT operations


Setting Up Airflow for dbt​

  1. Install Airflow (example using pip):
pip install apache-airflow
pip install apache-airflow-providers-postgres # or your database provider
  1. Define a DAG for dbt:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG(
'dbt_daily_pipeline',
start_date=datetime(2025, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:

dbt_run = BashOperator(
task_id='dbt_run',
bash_command='dbt run --profiles-dir /path/to/profiles --target prod'
)

dbt_test = BashOperator(
task_id='dbt_test',
bash_command='dbt test --profiles-dir /path/to/profiles --target prod'
)

dbt_run >> dbt_test

Explanation:

  • dbt_run → Executes the dbt run command
  • dbt_test → Executes tests after models are built
  • dbt_run >> dbt_test → Ensures tests run after models

Best Practices for dbt + Airflow​

Use separate DAGs for different workflows (e.g., staging vs production)
Keep tasks idempotent to avoid duplicate data
Enable logging and email alerts for failures
Schedule DAGs according to data availability and SLA
Use Airflow variables or connections for credentials instead of hardcoding


Monitoring dbt DAGs in Airflow​

  • Airflow UI provides visual DAG monitoring
  • Track task success, duration, and errors
  • Combine with Slack/email notifications for alerts