02. Orchestrating with Airflow
Orchestrating dbt with Airflow
Apache Airflow is a popular orchestration tool for scheduling and managing dbt workflows in production.
It allows you to define Directed Acyclic Graphs (DAGs) for automated execution of dbt tasks.
Why Use Airflow for dbt?
Schedule dbt runs with flexible intervals
Manage dependencies between models
Integrate notifications and monitoring
Combine dbt tasks with other ETL/ELT operations
Setting Up Airflow for dbt
- Install Airflow (example using pip):
pip install apache-airflow
pip install apache-airflow-providers-postgres # or your database provider
- Define a DAG for dbt:
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime
with DAG(
'dbt_daily_pipeline',
start_date=datetime(2025, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:
dbt_run = BashOperator(
task_id='dbt_run',
bash_command='dbt run --profiles-dir /path/to/profiles --target prod'
)
dbt_test = BashOperator(
task_id='dbt_test',
bash_command='dbt test --profiles-dir /path/to/profiles --target prod'
)
dbt_run >> dbt_test
Explanation:
dbt_run→ Executes thedbt runcommanddbt_test→ Executes tests after models are builtdbt_run >> dbt_test→ Ensures tests run after models
Best Practices for dbt + Airflow
Use separate DAGs for different workflows (e.g., staging vs production)
Keep tasks idempotent to avoid duplicate data
Enable logging and email alerts for failures
Schedule DAGs according to data availability and SLA
Use Airflow variables or connections for credentials instead of hardcoding
Monitoring dbt DAGs in Airflow
- Airflow UI provides visual DAG monitoring
- Track task success, duration, and errors
- Combine with Slack/email notifications for alerts