Schema Evolution with Iceberg
One of the most powerful features of Apache Iceberg is its native, reliable support for Schema Evolution.
In older data lake formats (like Hive), changing a table's schema often meant rewriting all the underlying data files or breaking downstream queries because columns were mapped by position. Iceberg solves this by mapping columns by a unique ID instead of their position or name.
Topics Covered
- Schema Evolution Concepts (Column IDs)
- Adding, Renaming, and Dropping Columns
- Type Promotion Rules
- Schema Evolution vs Partition Evolution
Schema Evolution Concepts
Under the hood, Iceberg assigns a unique integer ID to every column in a table. When you rename a column, the ID stays the same. When you drop a column, its ID is retired. When you add a new column, it gets a new ID.
Because Iceberg tracks these IDs in its metadata files, you can safely evolve schemas without ever needing to rewrite the existing Parquet data files.
Hands-On Demonstration: Evolving an Iceberg Table
Let's see how easy it is to evolve an Iceberg table in Snowflake. We will create an Iceberg table, add data, alter its schema, and observe how Iceberg handles it seamlessly.
1. Database & Schema Setup
Copy and paste the following code into your Snowflake worksheet.
USE ROLE SYSADMIN;
CREATE DATABASE IF NOT EXISTS BEEMS_ICEBERG_DB;
USE DATABASE BEEMS_ICEBERG_DB;
CREATE SCHEMA IF NOT EXISTS EVOLUTION_LAB;
USE SCHEMA EVOLUTION_LAB;
2. Creating the Initial Iceberg Table
Note: This assumes you have already set up an External Volume. If you don't have one, this syntax is just for conceptual demonstration.
-- Create an initial Iceberg table
CREATE OR REPLACE ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES (
CUST_ID INT,
FULL_NAME STRING,
CITY STRING
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'MY_ICEBERG_VOL'
BASE_LOCATION = 'profiles/';
3. Inserting Initial Data
Let's insert some mock customer data from around India.
INSERT INTO EVOLUTION_LAB.CUSTOMER_PROFILES VALUES
(1, 'Ananya Sharma', 'Delhi'),
(2, 'Vikram Singh', 'Chandigarh'),
(3, 'Rajesh Patil', 'Pune');
SELECT * FROM EVOLUTION_LAB.CUSTOMER_PROFILES;
4. Evolving the Schema: Adding & Dropping Columns
Now, the business requirements change. We no longer need the CITY column, but we desperately need to track the customer's STATE and PHONE_NUMBER.
In older data lakes, dropping a column would orphan data or break queries. With Iceberg, it's an instant metadata operation.
-- Drop a column
ALTER ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES DROP COLUMN CITY;
-- Add new columns
ALTER ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES ADD COLUMN STATE STRING;
ALTER ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES ADD COLUMN PHONE_NUMBER STRING;
Let's insert new data matching the new schema.
INSERT INTO EVOLUTION_LAB.CUSTOMER_PROFILES (CUST_ID, FULL_NAME, STATE, PHONE_NUMBER) VALUES
(4, 'Meera Menon', 'Kerala', '9876543210'),
(5, 'Karthik Natarajan', 'Tamil Nadu', '8765432109');
If we query the table now, Snowflake and Iceberg handle the merge automatically:
- The old records (Ananya, Vikram, Rajesh) will show
NULLfor State and Phone. - The old
CITYdata is no longer accessible via the query engine, but the physical data wasn't rewritten.
5. Evolving the Schema: Renaming & Type Promotion
Iceberg allows you to rename columns without rewriting data. It also allows Type Promotion—widening a data type safely (e.g., from INT to BIGINT, or FLOAT to DOUBLE).
-- Rename a column
ALTER ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES RENAME COLUMN FULL_NAME TO CUSTOMER_NAME;
-- Promote a type: Let's assume CUST_ID needs to hold much larger numbers now
ALTER ICEBERG TABLE EVOLUTION_LAB.CUSTOMER_PROFILES MODIFY COLUMN CUST_ID SET DATA TYPE NUMBER(38,0);
(To check who executed these schema changes, you can verify your active user profile by running SELECT CURRENT_USER();)
Schema Evolution vs Partition Evolution
- Schema Evolution: Changing the structure of the columns (Adding, Dropping, Renaming, Promoting).
- Partition Evolution: Changing how the table is partitioned (e.g., changing from partitioning by
Monthto partitioning byDay).
Iceberg supports both. Just like schema evolution, if you change an Iceberg table's partition scheme, the old data remains partitioned the old way, and new data is partitioned the new way. The query engine understands both transparently!
Backward and Forward Compatibility
Because Iceberg tracks schema changes through metadata snapshots:
- Backward Compatibility: You can read old data files using the new schema (missing columns are filled with NULLs).
- Time Travel (Forward Compatibility): If you use Time Travel to query the table as it looked yesterday, Iceberg uses yesterday's schema to read the data, ignoring columns added today.
Summary
- Iceberg maps columns by Unique IDs, not position or name.
- Adding, dropping, or renaming columns are instant metadata-only operations.
- Existing data files are never rewritten when evolving a schema.
- Iceberg also safely supports Type Promotion (widening types).