Skip to main content

Streams Tasks

πŸ“Œ What are Streams and Tasks in Snowflake?​

  • βœ… Streams: A stream is a change tracking mechanism in Snowflake. It’s like a CDC (Change Data Capture) pointer that tracks new rows, updates, and deletes in a table since the last time you queried the stream.

  • βœ… Tasks: A task is a scheduled or triggered job that runs SQL statements automatically in Snowflake. You can use tasks to process the changes tracked by streams.


πŸš€ Purpose​

πŸ”Ή Data optimization: Instead of re-scanning an entire table, streams let you only process incremental changes (e.g., new/updated rows). This is much more efficient!

πŸ”Ή Automation: Tasks let you automatically process those changes on a schedule or event trigger.


πŸ”§ How They Work Together​

1️⃣ A stream tracks changes in a source table. 2️⃣ A task reads from the stream and applies business logic or transformation to downstream tables.


πŸ’‘ Example Workflow​

Step 1: Create a Stream​

CREATE OR REPLACE STREAM my_table_stream 
ON TABLE my_table;

Step 2: Create a Task​

CREATE OR REPLACE TASK my_task
WAREHOUSE = my_wh
SCHEDULE = '5 MINUTE'
AS
INSERT INTO my_target_table
SELECT * FROM my_table_stream;

Step 3: Start the Task​

ALTER TASK my_task RESUME;

This automates incremental data processing – only new/changed rows are inserted into my_target_table!


βš™οΈ How This Supports Data Optimization​

βœ”οΈ Efficiency: You’re only processing deltas (changes), not entire tables.

βœ”οΈ Cost-effective: Fewer compute resources used for data transformation jobs.

βœ”οΈ Automation: No manual triggers needed – tasks run on a schedule or on-demand.


🏷️ Comparison: Streams/Tasks vs. Traditional ETL​

FeatureTraditional ETLStreams + Tasks in Snowflake
Change TrackingOften re-scans full tablesOnly tracks changes (deltas)
AutomationExternal tools (e.g., Airflow)Built directly into Snowflake
PerformanceCan be compute-heavyOptimized for delta-processing
ManagementMore complex pipeline managementSimpler & fully in-database

⚑ Key Takeaways​

βœ”οΈ Streams track changes automatically. βœ”οΈ Tasks let you automate the handling of these changes on a schedule. βœ”οΈ This combo is essential for incremental data processing, automated ETL, and near real-time data updates in Snowflake. βœ”οΈ Lower compute costs + less operational complexity.