Deployment vs Developer Credentials
Security and environment isolation are critical in Data Engineering. If a developer accidentally runs dbt run --full-refresh on the production database, it could cause hours of downtime and data loss.
To prevent this, dbt strictly separates Developer Credentials from Deployment Credentials.
Topics Covered
- Developer Credentials (Personal Dev Environments)
- Deployment Credentials (Production/Staging Environments)
- Service Accounts vs User Accounts
- Credential Security Best Practices
1. Developer Credentials (Personal Dev Environments)
When you write dbt code (either in the dbt Cloud IDE or locally using dbt Core), you are doing development work. You need a safe sandbox to test your changes without affecting production data.
Characteristics of Developer Credentials:
- Bound to a Human User: E.g.,
ARJUN_KUMAR@beems.tech. - Target Schema is Personal: When Arjun runs
dbt run, dbt automatically builds his tables in a personal development schema, usually named something likeDEV_ARJUN_KUMAR. - Authentication: Usually relies on Single Sign-On (SSO), Okta, or a personal username/password.
- Privileges: Can read from production raw data, but can only write to their personal dev schema.
(To verify your current Snowflake user, run SELECT CURRENT_USER();)
2. Deployment Credentials (Production Environments)
When code is merged to the main branch and deployed to production, a human is no longer executing the code. An automated scheduler (like dbt Cloud, Airflow, or GitHub Actions) is running the code.
Characteristics of Deployment Credentials:
- Bound to a Service Account: E.g.,
DBT_PROD_SVC_ACCOUNT. - Target Schema is Production: When the automated job runs, dbt builds the tables in the official production schemas, like
ANALYTICS_PROD. - Authentication: Uses Key-Pair Authentication (RSA tokens) or long-lived API tokens. Never a password!
- Privileges: Has write access to the production schemas.
3. How dbt Cloud Manages the Split
In dbt Cloud, you configure these credentials in completely different places to ensure separation of concerns.
Setting up the Deployment Environment
- A Data Engineering Lead goes to Environments -> New Environment.
- They select Deployment as the type.
- They enter the Service Account credentials (
DBT_PROD_SVC_ACCOUNT). - They define the target schema as
ANALYTICS_PROD. - Only administrators can view or modify these credentials.
Setting up Developer Environments
- When a new analyst (e.g., Priya) logs into dbt Cloud for the first time, she navigates to her Account Settings -> Credentials.
- She enters her personal Snowflake username and password.
- dbt automatically assigns her a default dev schema (e.g.,
DEV_PRIYA). - If she runs
dbt runin the IDE, the tables are built inDEV_PRIYA. She cannot physically build tables inANALYTICS_PRODfrom the IDE because her personal credentials lack the database privileges.
4. Security Best Practices for Service Accounts
Because the Deployment Service Account (DBT_PROD_SVC_ACCOUNT) has write access to production data, it is a highly sensitive credential.
Best Practices:
- Never use Passwords for Service Accounts: Passwords can be guessed or leaked. Use Key-Pair Authentication (RSA keys). In Snowflake, you generate a public/private key pair, assign the public key to the user, and paste the private key into dbt Cloud.
- Principle of Least Privilege: Do not grant the
ACCOUNTADMINrole to the dbt Service Account. Create a dedicated role (e.g.,DBT_PROD_ROLE) that only has the exact permissions needed to read the raw data and write to the analytics schemas. - Network Policies: Restrict the Service Account so it can only log in from dbt Cloud's specific IP addresses.
Summary
- Developer Credentials belong to humans. They write to isolated sandbox schemas (e.g.,
DEV_ARJUN) to safely test code. - Deployment Credentials belong to automated Service Accounts. They write to the official production schemas (e.g.,
ANALYTICS_PROD). - Never use a human's credentials for a production deployment job. If that person leaves the company and their account is deactivated, the production pipeline will instantly break.
- Secure your Service Accounts using Key-Pair Authentication.