Skip to main content

01. What is Schema Drift?

🔹 What is Schema Drift?​

Schema Drift is the phenomenon where the structure of incoming data changes unpredictably over time. In real-world data engineering, upstream applications evolve rapidly, and those changes eventually ripple down to your data pipelines.

If your data warehouse expects a rigid, fixed structure, schema drift will cause your ingestion pipelines to break.

When Does Schema Drift Happen?​

Schema drift typically occurs when:

  • Application Updates: Software engineers add a new feature to the product, resulting in a new column (e.g., user_preferences) being added to the source database.
  • Third-Party APIs: An external vendor updates their API payload and renames a field, or drops an old field entirely.
  • Human Error: Someone manually exports a CSV from Excel but rearranges the column order before sending it to the data team.

Things Associated with Schema Drift​

When dealing with schema drift, you must be prepared to handle:

  1. New Columns Added: The source data suddenly has more columns than your target table.
  2. Existing Columns Dropped: The source data is missing columns that your target table expects.
  3. Column Reordering: The columns arrive in a different order than before (common in CSVs).
  4. Data Type Changes: A column that used to send integer IDs (101) suddenly starts sending string hashes ("abc-101").

Snowflake provides several mechanisms to gracefully handle these changes without pipeline downtime, which we will explore in the next sections.