Environment Variables for Secrets
When we first set up our profiles.yml file, we hardcoded our Snowflake password directly into the file.
If you are just learning dbt on your personal laptop, this is fine. But in a real enterprise environment, hardcoding passwords is the #1 cause of security breaches. If you accidentally commit a hardcoded profiles.yml to GitHub, anyone in the world could log into your company's Snowflake account and steal customer data!
To secure your pipeline, you must use Environment Variables.
Topics Covered
- The
env_var()function - Updating
profiles.yml - Setting Environment Variables locally
- Setting Secrets in CI/CD (GitHub Actions)
1. The env_var() Function
dbt allows you to read environment variables directly from your operating system using the special Jinja function {{ env_var('VARIABLE_NAME') }}.
You can use this function in your profiles.yml to dynamically pull passwords at runtime without ever saving the password to disk.
2. Updating your profiles.yml
Let's assume our Data Engineer, Arjun, wants to secure his local connection to Snowflake.
First, he opens his ~/.dbt/profiles.yml file and replaces the hardcoded password with the env_var() function.
# ~/.dbt/profiles.yml
beems_retail:
target: dev
outputs:
dev:
type: snowflake
account: abc1234.ap-south-1.aws # AWS Mumbai Region
user: ARJUN_DEV
# REMOVE THIS:
# password: "MySuperSecretPassword123!"
# REPLACE WITH THIS:
password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
role: SYSADMIN
database: BEEMS_DEV
warehouse: COMPUTE_WH
schema: DEV_ARJUN
3. Setting Variables Locally
If Arjun types dbt run right now, dbt will throw an error: Env var required but not provided: 'DBT_SNOWFLAKE_PASSWORD'.
Arjun must tell his Mac/Linux terminal what that variable is before he runs dbt.
He can do this by exporting the variable in his terminal:
# 1. Set the variable in the current terminal session
export DBT_SNOWFLAKE_PASSWORD="MySuperSecretPassword123!"
# 2. Run dbt (dbt will successfully read the variable!)
dbt run
Making it Permanent (Local)
Typing export every time you open a terminal is annoying. Most developers place their secrets in a hidden .bashrc or .zshrc file on their laptop, which automatically loads the variables every time they open a terminal window.
# Add this to the bottom of ~/.zshrc
export DBT_SNOWFLAKE_PASSWORD="MySuperSecretPassword123!"
4. Setting Secrets in CI/CD
The real power of env_var() is in CI/CD orchestration.
When you deploy your project to production, you don't want a human typing in passwords. If you use GitHub Actions to orchestrate your dbt build, you will store your password in GitHub Repository Secrets.
In your GitHub Action YAML file, you simply pass the secure secret into the environment:
# .github/workflows/dbt_run.yml
jobs:
run-dbt:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Run dbt build
run: dbt build --target prod
# Pass the GitHub Secret into the environment variable!
env:
DBT_SNOWFLAKE_PASSWORD: ${{ secrets.PROD_SNOWFLAKE_PASSWORD }}
When GitHub Actions runs dbt build, dbt looks at the profiles.yml, sees the env_var() function, checks the environment, grabs the highly-secure password, and connects to Snowflake.
The password is never printed in logs and never saved in code!
Summary
- Never hardcode passwords in
profiles.ymlin an enterprise setting. - Use the
{{ env_var('VAR_NAME') }}function in yourprofiles.ymlfile. - Set the variable locally using
export VAR_NAME="secret". - Use this pattern to securely inject passwords via CI/CD orchestration tools like GitHub Actions or GitLab CI.