Failover & Failback Procedures
When a catastrophic cloud outage occurs (e.g., an AWS region goes completely offline), your primary Snowflake account becomes inaccessible. To maintain business continuity, the Administrator must execute a Failover to promote the secondary region to become the new primary.
Once the original region is restored, the Administrator must execute a Failback to return operations to normal.
Topics Covered
- Triggering a Failover
- Handling Application Connections (DNS)
- Estimating Data Loss (RPO)
- Executing a Failback
1. Triggering a Failover
A Failover is a manual, explicit action. Snowflake will not auto-failover for you, because doing so has massive implications for data ingestion and billing.
Let's assume our Primary account is AWS Mumbai and our Secondary replica is Azure Pune. AWS Mumbai has just gone completely offline.
To failover, the Administrator must log directly into the Secondary Account (Azure Pune) and execute the promotion command.
(To execute a failover, you must have the ACCOUNTADMIN role. To check your role, run SELECT CURRENT_ROLE();)
-- Step 1: Log into the Azure Pune Account via its specific URL
USE ROLE ACCOUNTADMIN;
-- Step 2: Promote the Failover Group to Primary
ALTER FAILOVER GROUP FG_PROD_CORE PRIMARY;
At this exact moment:
FG_PROD_COREin Azure Pune becomes the Primary.- The databases inside it transition from Read-Only to Read-Write.
- Any background replication schedules from Mumbai to Pune are immediately halted.
2. Handling Application Connections (DNS)
After failing over to Azure Pune, your data is available. However, your BI tools (Tableau), ETL pipelines (Airflow), and ingestion tools (Fivetran) are still pointing to the dead AWS Mumbai URL (org-aws_mumbai.snowflakecomputing.com).
How do you fix this?
- The Hard Way (Manual Updates): You must manually update the connection strings in Airflow, dbt, and Tableau to point to
org-azure_pune.snowflakecomputing.com. This is slow, error-prone, and dramatically increases your Recovery Time Objective (RTO). - The Easy Way (Client Redirect): If you purchased the Business Critical edition, you use Snowflake Client Redirect. You configure a global connection URL (e.g.,
my_company.global.snowflakecomputing.com). During a failover, you run a single SQL command to update the DNS routing.
-- Using Client Redirect to update the global DNS
ALTER CONNECTION MY_GLOBAL_CONNECTION PRIMARY;
Within roughly 60 seconds, all external applications will automatically reconnect to the new Azure Pune account without requiring any code changes!
3. Estimating Data Loss (RPO)
When you failover, you must accept that any data written to the Primary region after the last successful replication refresh is permanently lost.
If your replication schedule was set to 60 minutes, and the outage happened at 12:45 PM, any data written between 12:00 PM and 12:45 PM is gone from Snowflake. Your upstream systems (like Kafka or AWS SQS) must replay that missing 45 minutes of data into the new Azure Pune account.
4. Executing a Failback
Fast forward three days. AWS Mumbai is fully repaired and online. We want to return to our original architecture. This process is called Failback.
You cannot simply promote Mumbai back to Primary. During the three days we ran in Azure Pune, new data was written. We must first replicate that new data back to Mumbai.
The Failback Sequence
- Stop Ingestion: Pause your ETL tools and Snowpipes so no new data enters Azure Pune.
- Reverse the Replication: Log into AWS Mumbai (which is currently the Secondary) and pull the latest data from Azure Pune.
-- Run in AWS Mumbai
ALTER FAILOVER GROUP FG_PROD_CORE REFRESH; - Promote Mumbai: Once the refresh completes, promote Mumbai back to Primary.
-- Run in AWS Mumbai
ALTER FAILOVER GROUP FG_PROD_CORE PRIMARY; - Update DNS: Point the Client Redirect connection back to Mumbai.
-- Run in AWS Mumbai
ALTER CONNECTION MY_GLOBAL_CONNECTION PRIMARY; - Resume Ingestion: Turn your ETL pipelines and Snowpipes back on.
Operations are now fully restored to the original state!
Summary
- Failover promotes a read-only secondary replica to become a read-write primary.
- Always use Client Redirect (Global URLs) to avoid manually updating connection strings in your BI/ETL tools.
- Data written after the last successful replication refresh is lost and must be replayed by upstream systems.
- Failback requires explicitly refreshing the data backwards before promoting the original region.