Skip to main content

dbt Cloud API

While dbt Cloud has an excellent built-in scheduler, enterprise Data Engineering teams often use a master orchestrator (like Apache Airflow, Prefect, or Dagster) to control their entire pipeline.

For example, an Indian logistics company might have an Airflow DAG that:

  1. Triggers an AWS Lambda function to download files.
  2. Triggers Snowflake Snowpipe to ingest the files.
  3. Triggers dbt Cloud to transform the data.
  4. Triggers Tableau to refresh the dashboards.

To achieve step 3, Airflow must communicate with dbt Cloud programmatically. This is done using the dbt Cloud API.

Topics Covered​

  • Generating an API Service Token
  • Triggering a Job Programmatically
  • Polling for Job Completion
  • Downloading Run Artifacts

1. Generating an API Service Token​

Before you can make an API request, you must authenticate. You should never use a personal API token for an automated system. Instead, you create a Service Token.

  1. In dbt Cloud, navigate to Account Settings -> Service Tokens.
  2. Click New Token.
  3. Name the token (e.g., Airflow_Orchestrator).
  4. Assign the token the Job Admin permission (so it can trigger jobs).
  5. Copy the generated token immediately. It will not be shown again.

(To identify your own personal permissions in dbt Cloud, check your User Settings).

2. Triggering a Job Programmatically​

The most common API action is triggering a specific job to run. To do this, you need two pieces of information:

  • Your ACCOUNT_ID
  • The JOB_ID you want to run (You can find this in the URL when viewing the job in the dbt Cloud UI).

You send a POST request to the /run/ endpoint.

(Example using curl)

curl -X POST "https://cloud.getdbt.com/api/v2/accounts/12345/jobs/67890/run/" \
-H "Authorization: Token your_service_token_here" \
-H "Content-Type: application/json" \
-d '{"cause": "Triggered by Airflow upstream completion"}'

The API will respond with a JSON payload containing a unique Run ID (e.g., "id": 999111). You must save this Run ID for the next step.

3. Polling for Job Completion​

When you trigger a job via the API, the API immediately returns a 200 OK response. However, this only means the job started. It does not mean the job finished.

Airflow needs to wait until the dbt job finishes before triggering the Tableau refresh. To do this, Airflow must continuously "poll" the dbt Cloud API to check the status of the specific Run ID.

You send a GET request to the /runs/{run_id}/ endpoint.

curl -X GET "https://cloud.getdbt.com/api/v2/accounts/12345/runs/999111/" \
-H "Authorization: Token your_service_token_here"

The API will respond with a status integer:

  • 1 = Queued
  • 3 = Running
  • 10 = Success
  • 20 = Error
  • 30 = Cancelled

Your Python script (or Airflow Operator) should loop this request every 30 seconds until the status returns 10, 20, or 30.

4. Downloading Run Artifacts​

After a job runs, dbt generates JSON artifact files (like manifest.json and run_results.json). These files contain massive amounts of metadata about your project.

You can use the dbt Cloud API to download these artifacts for custom reporting or Data Observability tools (like Monte Carlo or Elementary).

# Download the manifest.json for a specific run
curl -X GET "https://cloud.getdbt.com/api/v2/accounts/12345/runs/999111/artifacts/manifest.json" \
-H "Authorization: Token your_service_token_here" \
-o manifest.json

Using Pre-Built Orchestrator Integrations​

If you are using a modern orchestrator, you rarely have to write these curl commands or Python polling loops manually.

  • Airflow: Use the DbtCloudRunJobOperator. It handles the triggering and polling automatically.
  • Dagster: Use the dbt_cloud_resource.
  • Prefect: Use the prefect-dbt task library.

You simply provide the API Token and the Job ID to the operator, and it handles the complex REST API interactions for you.

Summary​

  • The dbt Cloud API allows external orchestrators (like Airflow) to control dbt jobs.
  • Always use a Service Token for automated API calls, not a personal token.
  • Triggering a job returns a Run ID. You must poll that Run ID to determine when the job actually finishes.
  • Use native orchestrator integrations (like the DbtCloudRunJobOperator) whenever possible to avoid writing custom API polling logic.