02. dbt run-operation
Up until now, every macro we have written has been injected inside a SELECT statement within a .sql model file. The macro is executed when you type dbt run, and its only purpose is to help build that specific table.
Up until now, every macro we have written has been injected inside a SELECT statement within a .sql model file. The macro is executed when you type dbt run, and its only purpose is to help build that specific table.
When a Data Engineer joins a company, they rarely get to build a greenfield project from scratch. Usually, they are tasked with migrating legacy SQL procedures (e.g., thousands of lines of old Stored Procedures) into a clean, modular dbt project.
One of the most tedious parts of Data Engineering is writing repetitive boilerplate code.
If you write a Python function, you write a unit test for it. If you write a dbt SQL model, you write a data test for it (not_null, unique).
The DBT Cloud Alternative
A core principle of dbt is that it does not compute any data itself. dbt is essentially a highly intelligent SQL compiler and orchestrator. It writes the SQL, and then sends it to a data warehouse to do the actual heavy lifting.
️ What is Lineage in dbt?
A macro can be syntactically valid Jinja and still generate invalid or incorrect SQL. The reliable workflow is to inspect the rendered SQL and test representative inputs.
In dbt, the Directed Acyclic Graph (DAG) usually ends with your marts models.
As your dbt project grows, structuring your macros effectively becomes critical. In this section, we'll explore some common design patterns for advanced macro engineering.
In the previous section, we discussed why we override the target schema to prevent developers from overwriting each other's work while ensuring Production databases remain clean.
Defining Exposures in YAML creates beautiful documentation in the dbt Docs UI. However, the real power of Exposures is using them in the Command Line Interface (CLI) to perform Impact Analysis.
In the open-source version of dbt (dbt Core), you generate documentation by running dbt docs generate and hosting the static HTML site yourself.
For the last decade, the standard data architecture looked like this: