Skip to main content

Deployment vs Developer Credentials

Security and environment isolation are critical in Data Engineering. If a developer accidentally runs dbt run --full-refresh on the production database, it could cause hours of downtime and data loss.

To prevent this, dbt strictly separates Developer Credentials from Deployment Credentials.

Topics Covered​

  • Developer Credentials (Personal Dev Environments)
  • Deployment Credentials (Production/Staging Environments)
  • Service Accounts vs User Accounts
  • Credential Security Best Practices

1. Developer Credentials (Personal Dev Environments)​

When you write dbt code (either in the dbt Cloud IDE or locally using dbt Core), you are doing development work. You need a safe sandbox to test your changes without affecting production data.

Characteristics of Developer Credentials:

  • Bound to a Human User: E.g., ARJUN_KUMAR@beems.tech.
  • Target Schema is Personal: When Arjun runs dbt run, dbt automatically builds his tables in a personal development schema, usually named something like DEV_ARJUN_KUMAR.
  • Authentication: Usually relies on Single Sign-On (SSO), Okta, or a personal username/password.
  • Privileges: Can read from production raw data, but can only write to their personal dev schema.

(To verify your current Snowflake user, run SELECT CURRENT_USER();)

2. Deployment Credentials (Production Environments)​

When code is merged to the main branch and deployed to production, a human is no longer executing the code. An automated scheduler (like dbt Cloud, Airflow, or GitHub Actions) is running the code.

Characteristics of Deployment Credentials:

  • Bound to a Service Account: E.g., DBT_PROD_SVC_ACCOUNT.
  • Target Schema is Production: When the automated job runs, dbt builds the tables in the official production schemas, like ANALYTICS_PROD.
  • Authentication: Uses Key-Pair Authentication (RSA tokens) or long-lived API tokens. Never a password!
  • Privileges: Has write access to the production schemas.

3. How dbt Cloud Manages the Split​

In dbt Cloud, you configure these credentials in completely different places to ensure separation of concerns.

Setting up the Deployment Environment​

  1. A Data Engineering Lead goes to Environments -> New Environment.
  2. They select Deployment as the type.
  3. They enter the Service Account credentials (DBT_PROD_SVC_ACCOUNT).
  4. They define the target schema as ANALYTICS_PROD.
  5. Only administrators can view or modify these credentials.

Setting up Developer Environments​

  1. When a new analyst (e.g., Priya) logs into dbt Cloud for the first time, she navigates to her Account Settings -> Credentials.
  2. She enters her personal Snowflake username and password.
  3. dbt automatically assigns her a default dev schema (e.g., DEV_PRIYA).
  4. If she runs dbt run in the IDE, the tables are built in DEV_PRIYA. She cannot physically build tables in ANALYTICS_PROD from the IDE because her personal credentials lack the database privileges.

4. Security Best Practices for Service Accounts​

Because the Deployment Service Account (DBT_PROD_SVC_ACCOUNT) has write access to production data, it is a highly sensitive credential.

Best Practices:

  1. Never use Passwords for Service Accounts: Passwords can be guessed or leaked. Use Key-Pair Authentication (RSA keys). In Snowflake, you generate a public/private key pair, assign the public key to the user, and paste the private key into dbt Cloud.
  2. Principle of Least Privilege: Do not grant the ACCOUNTADMIN role to the dbt Service Account. Create a dedicated role (e.g., DBT_PROD_ROLE) that only has the exact permissions needed to read the raw data and write to the analytics schemas.
  3. Network Policies: Restrict the Service Account so it can only log in from dbt Cloud's specific IP addresses.

Summary​

  • Developer Credentials belong to humans. They write to isolated sandbox schemas (e.g., DEV_ARJUN) to safely test code.
  • Deployment Credentials belong to automated Service Accounts. They write to the official production schemas (e.g., ANALYTICS_PROD).
  • Never use a human's credentials for a production deployment job. If that person leaves the company and their account is deactivated, the production pipeline will instantly break.
  • Secure your Service Accounts using Key-Pair Authentication.