Prefect & Dagster Integrations
For many years, Apache Airflow was the undisputed king of Data Orchestration. However, modern Data Engineering teams are increasingly adopting newer orchestrators like Prefect and Dagster because they offer better developer experiences, easier local testing, and native awareness of the data itself.
Both tools have first-class integrations with dbt.
Topics Covered
- Why use an external orchestrator?
- Prefect: The Task-Oriented Approach
- Dagster: The Asset-Oriented Approach
1. Why use an external orchestrator?
dbt Cloud has a great built-in scheduler. Why complicate your stack with Prefect or Dagster?
Because dbt only handles the "T" (Transform) in ELT. If an Indian fintech company needs to pull API data from the National Payments Corporation of India (NPCI) into Snowflake before dbt runs, the dbt Cloud scheduler cannot execute that Python extraction script. You need a master orchestrator that says:
- Run Python API Extraction
- If successful, trigger dbt Cloud
- If dbt Cloud is successful, trigger a Machine Learning pipeline
2. Prefect Integration (Task-Oriented)
Prefect is a modern, Pythonic orchestrator. You write standard Python functions, add a @task decorator, and Prefect handles the scheduling, retries, and logging.
Prefect integrates with dbt using the prefect-dbt library.
Example: Triggering a dbt Cloud Job from Prefect
from prefect import flow
from prefect_dbt.cloud import DbtCloudJobRun
# Initialize the task block using a pre-configured dbt Cloud API Token
dbt_cloud_job = DbtCloudJobRun(
dbt_cloud_credentials="my-dbt-cloud-credentials-block",
job_id=12345
)
@flow(name="Daily Finance Pipeline")
def run_finance_pipeline():
print("Step 1: Extracting data from NPCI APIs...")
# extract_npci_data()
print("Step 2: Triggering dbt Cloud to transform the data...")
# The Prefect task automatically hits the dbt Cloud API, triggers the job,
# and polls the status until it succeeds or fails.
result = dbt_cloud_job.run()
print(f"dbt Job finished with status: {result}")
if __name__ == "__main__":
run_finance_pipeline()
3. Dagster Integration (Asset-Oriented)
Dagster represents a fundamental shift in orchestration. Instead of focusing on Tasks ("Run the script"), Dagster focuses on Software-Defined Assets ("Ensure the DIM_CUSTOMERS table exists and is up-to-date").
Because dbt is inherently asset-oriented (every .sql file represents a table/asset), Dagster and dbt are a perfect match.
Using the dagster-dbt library, Dagster reads your dbt project's manifest.json file. It automatically translates every single dbt model into a Dagster Asset.
Example: Loading dbt models into Dagster
import os
from dagster import Definitions
from dagster_dbt import DbtCliResource, load_assets_from_dbt_project
# 1. Define where your dbt project lives
DBT_PROJECT_DIR = "/opt/dbt_projects/finance_project"
# 2. Define the dbt CLI resource
dbt_resource = DbtCliResource(project_dir=DBT_PROJECT_DIR)
# 3. Dagster magically reads the manifest and creates an Asset for every model!
dbt_assets = load_assets_from_dbt_project(
project_dir=DBT_PROJECT_DIR,
)
# 4. Register the definitions
defs = Definitions(
assets=[dbt_assets],
resources={"dbt": dbt_resource},
)
The Dagster UI Advantage
Because Dagster understands dbt at the model level (not just the job level), the Dagster Web UI will literally display your dbt DAG (Lineage Graph).
If stg_payments fails, Dagster knows exactly which downstream models (fct_daily_revenue) to skip, and exactly which upstream Python extraction tasks caused the failure. You get complete end-to-end data lineage from Python to dbt in a single UI.
Summary
- External orchestrators are required when your pipeline includes non-dbt steps (like Python API extraction or Machine Learning).
- Prefect uses a Pythonic, task-based approach. Use the
prefect-dbtlibrary to easily trigger dbt Cloud jobs. - Dagster uses a Software-Defined Asset approach. It deeply integrates with dbt Core, reading the
manifest.jsonto display full end-to-end lineage inside the Dagster UI.