Skip to main content

๐Ÿ”„ Separation of Storage & Compute in Snowflake

๐Ÿ“Œ Overviewโ€‹

One of the most significant architectural innovations of Snowflake is its complete separation of storage and compute. Unlike traditional database systems that tightly couple compute and storage, Snowflake's cloud-native design allows both layers to operate independently, enabling elastic scalability, cost efficiency, and unparalleled concurrency.


๐Ÿ’ก Why Separation Mattersโ€‹

โŒ Traditional Architecturesโ€‹

CharacteristicDrawback
Tightly coupled compute-storageCannot scale independently โ€” over-provisioning required
Shared compute across usersResource contention under high concurrency
Always-on computeIncur costs even when idle

In traditional systems, adding more storage means adding more compute (and vice versa), leading to inefficiency, rigid scaling, and high costs.


โœ… Snowflakeโ€™s Modern Approachโ€‹

Snowflake decouples storage and compute to allow on-demand scaling, independent provisioning, and multi-workload processing โ€” all without performance trade-offs.


๐Ÿ’พ Storage Layer โ€“ Centralized and Cloud-Nativeโ€‹

The storage layer is a centralized repository where Snowflake securely stores your data in optimized formats.

๐Ÿ”น Key Features:โ€‹

  • Supports structured and semi-structured data (CSV, JSON, Avro, Parquet, ORC)
  • Data is compressed, encrypted, and optimized
  • Hosted on cloud providers like AWS S3, Azure Blob, or GCP Cloud Storage
  • Always-on and durable, regardless of compute activity
  • Acts as a single source of truth for all workloads

๐Ÿ“‚ Think of it as a central data lake accessible by any compute cluster, any time.


โš™๏ธ Compute Layer โ€“ Virtual Warehousesโ€‹

The compute layer is made up of Virtual Warehouses โ€” independent compute clusters that execute queries, run transformations, and load/export data.

๐Ÿ”น Key Features:โ€‹

  • Isolated compute resources for different users or teams
  • Start, stop, resize, or scale-out based on workload demands
  • No shared CPU/memory between warehouses โ€” eliminates performance contention
  • Support for multi-cluster auto-scaling to handle concurrency
  • Compute is billed only when active, minimizing idle costs

๐Ÿ› ๏ธ Virtual Warehouses act like data workers who process the information stored in your central warehouse shelves.


๐Ÿ”„ How They Work Togetherโ€‹

ComponentFunction
StorageCentralized location for all raw and processed data
Compute (VW)Pulls from storage to run queries, transform, and load
  • Storage remains persistent and shared
  • Virtual Warehouses can operate independently or concurrently
  • There is no duplication or latency penalty across compute clusters

๐ŸŽฏ Benefits of Separationโ€‹

BenefitDescription
Independent ScalingScale storage or compute independently, per requirement
High ConcurrencyRun multiple workloads without impacting performance
Cost EfficiencyPay for compute only when used โ€” storage remains low-cost
Resource IsolationAvoid cross-team interference via dedicated VWs
Operational FlexibilityEasily suspend, resume, or scale compute on demand
Simplified ArchitectureEasier to maintain, monitor, and allocate resources

๐Ÿ‘ฅ Real-World Exampleโ€‹

Imagine three teams working on the same data source:

TeamUse CaseVirtual Warehouse
Data EngineeringETL Pipelineetl_wh (Large)
BI TeamDashboard Queriesbi_wh (Medium)
Data ScienceML Model Trainingds_wh (XL, Autoscale)

Each team:

  • Accesses the same centralized storage
  • Operates with no impact on othersโ€™ performance
  • Pays for compute only when used
  • Gets customized compute for their specific workload

๐Ÿง  Final Summary: Storage vs Compute in Snowflake (Transposed View)โ€‹

Feature๐Ÿ“ฆ Storageโš™๏ธ Virtual Warehouse (Compute)
RoleStores structured and semi-structured data persistentlyExecutes queries, ETL jobs, transformations, and model training
AvailabilityAlways available โ€” cloud-native and always-onOn-demand โ€” can be started, stopped, and scaled dynamically
Billing ModelLow, usage-based pricing (storage volume per month)Pay-per-use, per-second billing while active
ScalabilityScales automatically and independently without affecting computeScales vertically (size) and horizontally (multi-cluster) as needed
Performance ModelPassive layer โ€” does not impact performance directlyPerformance varies by warehouse size, concurrency model, and scaling configuration
CollaborationShared across all teams โ€” single source of truthIsolated per team or workload โ€” no contention or interference
AnalogyLike a data library โ€” centralized, organized, and persistentLike workers who read, write, and process books from the library independently

๐Ÿ”‘ Takeaway: Snowflakeโ€™s separation of storage and compute allows teams to scale, collaborate, and optimize costs โ€” all while maintaining high performance and operational simplicity.


โœ… What Does โ€œFully Managed by Snowflakeโ€ Mean?โ€‹

When we say "Fully Managed by Snowflake", it means Snowflake takes care of everything related to data storage, compute, security, optimization, and availability โ€” so you don't have to manage infrastructure or low-level configurations.

Think of Snowflake as "Data Warehouse-as-a-Service" โ€” like Google Docs for data warehouses: You use it, collaborate, and focus on content. Snowflake takes care of servers, file systems, and upgrades.

๐Ÿ“Š Summary Table โ€“ Architecture Componentsโ€‹

LayerResponsibilityFully Managed Capabilities
Storage LayerData storage (structured/semi-structured)Encrypted, compressed, automatically optimized and abstracted โ€” no manual configs
Compute LayerData processing (queries, loads, transforms)Auto-scalable virtual warehouses, serverless provisioning, pause/resume on demand
Services LayerMetadata, security, orchestration, optimizationAuth, query optimization, metadata, time travel, fail-safe โ€” always-on and auto-managed