External Iceberg Tables — External Volumes & Catalog Integration
In Snowflake, an Iceberg Table allows you to query data stored in open formats (Apache Parquet) using the Apache Iceberg table specification.
When we talk about "External" Iceberg tables, we mean that both the metadata and the data files are managed by an external system (like Apache Spark, AWS Athena, or an external catalog like AWS Glue or Polaris), and Snowflake is simply reading them.
Topics Covered
- External Volume Configuration
- Catalog Integration
- Creating External Iceberg Tables
- Refreshing Metadata
- Interoperability
The Components of an External Iceberg Table
To connect Snowflake to an external Iceberg table, you need two underlying objects:
- External Volume: Tells Snowflake where the data physical files (Parquet) and metadata files (JSON/Avro) are located in cloud storage (S3, GCS, Azure).
- Catalog Integration: Tells Snowflake how to find the latest metadata pointer. Iceberg uses a catalog to track the absolute latest snapshot of a table.
Hands-On Demonstration: Integrating an External Iceberg Table
In this lab, we will simulate connecting Snowflake to an Iceberg table that was created and is actively being written to by an Apache Spark job running in AWS.
1. Database & Schema Setup
Copy and paste the following code into your Snowflake worksheet.
USE ROLE SYSADMIN;
CREATE DATABASE IF NOT EXISTS BEEMS_DATALAKE_DB;
USE DATABASE BEEMS_DATALAKE_DB;
CREATE SCHEMA IF NOT EXISTS EXTERNAL_CATALOG_SCHEMA;
USE SCHEMA EXTERNAL_CATALOG_SCHEMA;
2. Creating the External Volume
First, Snowflake needs permission to read the bucket where the Parquet files live.
(Note: If you are setting this up in a real environment, you need ACCOUNTADMIN or a role with CREATE EXTERNAL VOLUME. You can check your user identity by running SELECT CURRENT_USER();)
-- Assume you have the ACCOUNTADMIN role
USE ROLE ACCOUNTADMIN;
CREATE OR REPLACE EXTERNAL VOLUME MY_S3_ICEBERG_VOL
STORAGE_LOCATIONS =
(
(
NAME = 'my-s3-us-east-1'
STORAGE_PROVIDER = 'S3'
STORAGE_BASE_URL = 's3://beems-datalake-bucket/iceberg-data/'
STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/my-snowflake-role'
)
);
3. Creating the Catalog Integration
Next, Snowflake needs to know where the catalog is. For this example, we'll assume AWS Glue is the catalog tracking the Iceberg table.
CREATE OR REPLACE CATALOG INTEGRATION GLUE_CATALOG_INT
CATALOG_SOURCE = GLUE
CATALOG_NAMESPACE = 'default'
TABLE_FORMAT = ICEBERG
GLUE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/my-glue-role'
GLUE_CATALOG_ID = '123456789012'
GLUE_REGION = 'us-east-1'
ENABLED = TRUE;
4. Creating the Unmanaged Iceberg Table
Now we tie the Volume and the Catalog together to create the table in Snowflake. Because AWS Glue manages this table, it is Read-Only from Snowflake's perspective.
USE ROLE SYSADMIN;
CREATE OR REPLACE ICEBERG TABLE EXTERNAL_CATALOG_SCHEMA.SPARK_ETL_CUSTOMERS
EXTERNAL_VOLUME = 'MY_S3_ICEBERG_VOL'
CATALOG = 'GLUE_CATALOG_INT'
CATALOG_TABLE_NAME = 'spark_etl_customers';
5. Querying the External Data
Even though Snowflake didn't create the data, you can query it with standard SQL. Let's look at the customer data that Spark wrote into the data lake.
-- Querying data created by Spark/Trino
SELECT * FROM EXTERNAL_CATALOG_SCHEMA.SPARK_ETL_CUSTOMERS;
(Sample output you might see, featuring data from across India)
| CUST_ID | NAME | STATE | LIFETIME_VALUE |
|---|---|---|---|
| 1001 | Rohan Sharma | Delhi | 45000.00 |
| 1002 | Kavya Pillai | Kerala | 120500.50 |
| 1003 | Siddharth Patel | Gujarat | 8900.00 |
| 1004 | Anjali Menon | Karnataka | 67000.75 |
6. Refreshing Metadata (If not using automated integration)
When AWS Glue updates the table (e.g., Spark writes a new batch of data), Snowflake needs to know. If you configure automated refresh using Cloud Notifications (SQS/EventBridge), Snowflake updates instantly.
If you haven't configured automated notifications, you must manually refresh the metadata so Snowflake can see the latest snapshot:
-- Manually fetch the latest Iceberg snapshot from Glue
ALTER ICEBERG TABLE EXTERNAL_CATALOG_SCHEMA.SPARK_ETL_CUSTOMERS REFRESH;
Interoperability with Spark, Trino, and Flink
The true power of Iceberg is compute engine interoperability.
- A Flink streaming job can continuously write raw events to the S3 bucket.
- A Spark job can run complex ML transformations and update the Iceberg metadata.
- Snowflake can immediately query the exact same files for business intelligence dashboards.
- There is zero data movement or copying required between these engines.
Summary
- External Iceberg Tables are managed by an external catalog (Glue, Polaris, REST) and are Read-Only in Snowflake.
- You must configure an External Volume (for storage access) and a Catalog Integration (for metadata tracking).
- They enable seamless interoperability between Snowflake and open-source ecosystems (Spark, Flink, Trino).