Open Table Format Concepts (Iceberg vs Delta vs Hudi)
Historically, data architectures fell into two distinct camps: Data Warehouses and Data Lakes.
- Data Warehouses (like older versions of Snowflake, Redshift, Teradata) provided ACID transactions and fast SQL, but locked your data into a proprietary format.
- Data Lakes (S3, GCS) held massive amounts of cheap data in open formats like Parquet, but lacked reliability, DML (Updates/Deletes), and fast performance.
Open Table Formats bridge this gap, creating the Lakehouse architecture. They bring database-like features (ACID transactions, Time Travel, Schema Evolution) to data lakes while keeping the data in open, non-proprietary file formats.
Topics Covered
- Why Open Formats Matter
- Apache Iceberg Architecture
- Delta Lake & Apache Hudi Overviews
- Iceberg vs Delta vs Hudi Comparison
Why Open Formats Matter (Vendor Independence)
If you load your data into a traditional data warehouse, that vendor owns your storage. If you want to use a different compute engine (like Apache Spark or an AI tool) to analyze that data, you must copy it out of the warehouse.
With Open Table Formats, you own your storage. The data sits in your S3/GCS bucket as open Parquet files. Multiple compute engines (Snowflake, Spark, Trino, Flink) can all read and write to the exact same dataset simultaneously without copying data. This prevents vendor lock-in.
Apache Iceberg Architecture
Apache Iceberg is the open table format championed by Snowflake. Iceberg tracks tables at the file level, not the folder level (like Hive).
An Iceberg table consists of three layers:
- Data Layer: The actual data stored in Apache Parquet (or ORC) files.
- Metadata Layer: Contains Manifest Files (lists the Parquet files and their min/max stats) and Metadata JSON files (tracks table schema, partition spec, and snapshots).
- Catalog: A centralized registry (like Snowflake, AWS Glue, or Polaris) that points to the latest Metadata JSON file. This ensures atomic commits and ACID compliance.
Hands-On Demonstration: The Independence of Open Formats
Let's simulate a scenario where a data engineer creates an Iceberg table, and we query it to prove that the data is not locked in a proprietary system.
1. Database & Schema Setup
Copy and paste the following code into your Snowflake worksheet.
USE ROLE SYSADMIN;
CREATE DATABASE IF NOT EXISTS BEEMS_CONCEPTS_DB;
USE DATABASE BEEMS_CONCEPTS_DB;
CREATE SCHEMA IF NOT EXISTS FORMAT_LAB;
USE SCHEMA FORMAT_LAB;
2. Conceptual Iceberg Table Creation
We will create a managed Iceberg table tracking university admissions across India.
(Note: Assumes MY_ICEBERG_VOL exists. Treat as conceptual if your sandbox lacks external volumes).
CREATE OR REPLACE ICEBERG TABLE FORMAT_LAB.UNIVERSITY_ADMISSIONS (
APPLICATION_ID INT,
STUDENT_NAME STRING,
STATE STRING,
STATUS STRING
)
CATALOG = 'SNOWFLAKE'
EXTERNAL_VOLUME = 'MY_ICEBERG_VOL'
BASE_LOCATION = 'admissions/';
-- Insert mock data
INSERT INTO FORMAT_LAB.UNIVERSITY_ADMISSIONS VALUES
(5001, 'Anjali Menon', 'Kerala', 'Accepted'),
(5002, 'Rohan Sharma', 'Delhi', 'Waitlisted'),
(5003, 'Priya Banerjee', 'West Bengal', 'Accepted'),
(5004, 'Siddharth Patel', 'Gujarat', 'Rejected'),
(5005, 'Karthik Natarajan', 'Tamil Nadu', 'Accepted');
SELECT * FROM FORMAT_LAB.UNIVERSITY_ADMISSIONS;
3. Proving Independence
Because this is an Iceberg table, the data is stored in your S3 bucket. If you wanted to, you could open an Apache Spark notebook (e.g., in Databricks or EMR) and run this exact query against the exact same data without moving it:
# Spark PySpark equivalent query (Conceptual)
# df = spark.read.format("iceberg").load("snowflake_catalog.BEEMS_CONCEPTS_DB.FORMAT_LAB.UNIVERSITY_ADMISSIONS")
# df.show()
(Reminder: If you need to verify which user you are currently logged in as while running these scripts, you can run SELECT CURRENT_USER();)
Delta Lake & Apache Hudi Overview
While Iceberg is heavily supported by Snowflake, it has two main competitors in the open table format space.
Delta Lake
Developed primarily by Databricks, Delta Lake uses a transaction log (a _delta_log directory) composed of JSON files to track which Parquet files belong to the table. It is highly optimized for Spark and tightly integrated into the Databricks ecosystem. Snowflake can read Delta tables, but its native support is strongest for Iceberg.
Apache Hudi
Originally developed at Uber, Hudi stands for "Hadoop Upserts Deletes and Incrementals". It is heavily optimized for streaming workloads and near real-time ingestion. It supports two table types: Copy-on-Write (like Iceberg/Delta) and Merge-on-Read (optimized for high-frequency updates).
Comparison: Iceberg vs Delta vs Hudi
| Feature | Apache Iceberg | Delta Lake | Apache Hudi |
|---|---|---|---|
| Origin | Netflix / Apple | Databricks | Uber |
| Best For | Large scale analytic queries, multi-engine interoperability | Databricks ecosystem, Spark-centric workloads | High-frequency streaming and upserts |
| Metadata Tracking | Distributed metadata tree | Central transaction log directory | Timeline/Commit log |
| Snowflake Support | First-class citizen (Read/Write/Managed) | External Read-only | External Read-only |
Summary
- Open Table Formats (Iceberg, Delta, Hudi) separate compute from storage, preventing vendor lock-in.
- Apache Iceberg is Snowflake's chosen format for Lakehouse architecture, offering full Read/Write and managed capabilities.
- The metadata layer is what provides database-like features (ACID, Time Travel) on top of standard Parquet files in your data lake.