01. What is Schema Drift?
🔹 What is Schema Drift?​
Schema Drift is the phenomenon where the structure of incoming data changes unpredictably over time. In real-world data engineering, upstream applications evolve rapidly, and those changes eventually ripple down to your data pipelines.
If your data warehouse expects a rigid, fixed structure, schema drift will cause your ingestion pipelines to break.
When Does Schema Drift Happen?​
Schema drift typically occurs when:
- Application Updates: Software engineers add a new feature to the product, resulting in a new column (e.g.,
user_preferences) being added to the source database. - Third-Party APIs: An external vendor updates their API payload and renames a field, or drops an old field entirely.
- Human Error: Someone manually exports a CSV from Excel but rearranges the column order before sending it to the data team.
Things Associated with Schema Drift​
When dealing with schema drift, you must be prepared to handle:
- New Columns Added: The source data suddenly has more columns than your target table.
- Existing Columns Dropped: The source data is missing columns that your target table expects.
- Column Reordering: The columns arrive in a different order than before (common in CSVs).
- Data Type Changes: A column that used to send integer IDs (
101) suddenly starts sending string hashes ("abc-101").
Snowflake provides several mechanisms to gracefully handle these changes without pipeline downtime, which we will explore in the next sections.