Skip to main content

Batch vs Streaming Architecture

In data engineering, the way data moves from source to destination fundamentally dictates how fresh the data is when it reaches the business user. The two primary paradigms are Batch Processing and Stream Processing.

Batch Processing Fundamentals​

Batch processing involves collecting data over a period of time (or volume), and then processing it all at once in a large "batch".

  • Pros: Highly efficient for massive datasets, easier to manage, less complex infrastructure.
  • Cons: High latency. Data is only as fresh as the last batch run (e.g., daily or hourly).

Stream Processing Fundamentals​

Stream processing (or Real-Time processing) involves processing data continuously as soon as it is generated or received.

  • Pros: Extremely low latency. Data is available for analysis in sub-seconds or seconds.
  • Cons: More complex infrastructure (Kafka, Flink, Streaming APIs), harder to manage out-of-order data, generally more expensive.

Lambda vs Kappa Architecture​

As architectures evolved to handle both paradigms, two famous design patterns emerged:

1. Lambda Architecture​

Maintains two separate data paths: a Batch layer (comprehensive, accurate, but slow) and a Speed layer (fast, streaming, but potentially less accurate). A Serving layer merges the results for the end-user.

  • Drawback: You have to maintain the same logic twice (e.g., once in Spark for batch, once in Flink for streaming).

2. Kappa Architecture​

Simplifies the pipeline by treating everything as a stream. The batch layer is entirely removed. If you need to reprocess historical data, you simply replay the streaming log (like Kafka) from the beginning. Snowflake's Streaming API strongly supports Kappa-like architectures.

Hands-On Demonstration: Simulating Batch vs Streaming​

Let's simulate how data arrives in a Batch scenario versus a Streaming scenario in Snowflake.

1. Database & Schema Setup​

Copy and paste the following code into your Snowflake worksheet.

USE ROLE SYSADMIN;

CREATE DATABASE IF NOT EXISTS BEEMS_ARCHITECTURE_DB;
USE DATABASE BEEMS_ARCHITECTURE_DB;

CREATE SCHEMA IF NOT EXISTS PATTERN_LAB;
USE SCHEMA PATTERN_LAB;

2. Creating the Tables​

-- The target table for Batch data
CREATE OR REPLACE TABLE PATTERN_LAB.BATCH_ORDERS (
ORDER_ID INT,
CUSTOMER_NAME STRING,
STATE STRING,
ORDER_TOTAL NUMBER(10,2),
PROCESSED_AT TIMESTAMP_NTZ
);

-- The target table for Streaming data
CREATE OR REPLACE TABLE PATTERN_LAB.STREAMING_ORDERS (
ORDER_ID INT,
CUSTOMER_NAME STRING,
STATE STRING,
ORDER_TOTAL NUMBER(10,2),
PROCESSED_AT TIMESTAMP_NTZ
);

3. Simulating Batch Data Ingestion​

In a batch process, all orders for the hour or day arrive at once via a scheduled task (like Airflow or an hourly Snowflake Task).

-- Simulating a 12:00 PM Batch Run (All at once)
INSERT INTO PATTERN_LAB.BATCH_ORDERS VALUES
(101, 'Kavya Pillai', 'Kerala', 450.00, '2023-10-01 12:00:00'),
(102, 'Rohan Sharma', 'Delhi', 1200.50, '2023-10-01 12:00:00'),
(103, 'Aarti Desai', 'Gujarat', 85.00, '2023-10-01 12:00:00'),
(104, 'Manoj Kumar', 'Bihar', 320.00, '2023-10-01 12:00:00'),
(105, 'Srinath Reddy', 'Telangana', 950.00, '2023-10-01 12:00:00');

4. Simulating Streaming Data Ingestion​

In a streaming process, orders arrive individually, exactly when the user clicks "Checkout" on the website.

-- Arrives at 12:01:05 PM
INSERT INTO PATTERN_LAB.STREAMING_ORDERS VALUES (106, 'Pooja Bhatt', 'Maharashtra', 110.00, '2023-10-01 12:01:05');

-- Arrives at 12:01:42 PM
INSERT INTO PATTERN_LAB.STREAMING_ORDERS VALUES (107, 'Vikram Singh', 'Punjab', 4500.00, '2023-10-01 12:01:42');

-- Arrives at 12:02:15 PM
INSERT INTO PATTERN_LAB.STREAMING_ORDERS VALUES (108, 'Anjali Menon', 'Karnataka', 210.00, '2023-10-01 12:02:15');

Snowflake's Approach to Real-Time​

Historically, Snowflake was a Batch-only Data Warehouse. Today, Snowflake bridges the gap:

  • Snowpipe (File-based): Provides "micro-batching" (1-minute latency).
  • Snowpipe Streaming API: Provides true streaming (sub-second latency) directly into tables.
  • Dynamic Tables: Automates the continuous transformation of streaming data, acting as a declarative alternative to complex streaming engines like Flink.

(To ensure you have the SYSADMIN role required to experiment with these architectures, you can check your active user profile. Run SELECT CURRENT_USER(); to see your current logged-in user).

Choosing Batch vs Streaming​

ScenarioRecommendation
Fraud Detection (Blocking a transaction instantly)Streaming
End-of-Month Financial ReportingBatch
Real-Time Logistics TrackingStreaming
Nightly Machine Learning Model RetrainingBatch

Rule of Thumb: Default to Batch unless the business explicitly requires Real-Time data to make immediate operational decisions. Streaming is more complex and expensive.

Summary​

  • Batch processes data in chunks; Streaming processes data as it arrives.
  • Lambda Architecture uses dual paths; Kappa Architecture treats everything as a stream.
  • Snowflake supports both paradigms, allowing you to use SQL for continuous transformations.