🚀 Deploying Models & Monitoring in Snowflake
Snowflake’s Snowpark API enables data engineers and data scientists to build and deploy ML models within Snowflake. Let’s break down the deployment and monitoring aspects:
1️⃣ Typical Deployment Process
-
✅ 1. Model training:
Models are trained outside Snowflake (locally, in notebooks, or in a pipeline) using libraries likescikit-learnorxgboost. -
✅ 2. Model serialization:
Save the trained model as a.pkl(or similar) file:
import joblib
joblib.dump(model, 'my_model.pkl')
- ✅ 3. Upload model to Snowflake stage:
Use the Snowflake
PUTcommand or UI to upload the file:
PUT file://my_model.pkl @my_stage;
- ✅ 4. Model inference script: Use a Snowpark script or stored procedure to load the model and make predictions:
import joblib
import pandas as pd
from snowflake.snowpark import Session
def predict(session: Session) -> str:
# Load the data
df = session.table("CUSTOMER_DATA").to_pandas()
# Load the model from stage
session.file.get("@my_stage/my_model.pkl", "my_model.pkl")
model = joblib.load("my_model.pkl")
# Make predictions
predictions = model.predict(df[["AGE", "INCOME"]])
df["PREDICTION"] = predictions
# Save predictions back to Snowflake
session.create_dataframe(df).write.mode("overwrite").save_as_table("PREDICTIONS")
return "Predictions saved to table!"
- ✅ 5. Register as a Stored Procedure: Deploy this as a stored procedure in Snowflake so it can run server-side.
2️⃣ Automated Deployment via CI/CD
You can automate model deployment and updates using CI/CD tools (like GitHub Actions or Azure DevOps Pipelines). A typical workflow:
- ✅ Train the model (locally or via pipeline)
- ✅ Push updated
.pklfile to a Snowflake stage - ✅ Run a pipeline step to trigger the stored procedure for inference
This ensures consistent and automated updates of your ML workflows!
3️⃣ Monitoring Model Performance
Monitoring is essential for understanding model health and data drift.
-
✅ Metrics you might track:
- Prediction accuracy (e.g., using actuals vs. predictions stored in Snowflake)
- Model version (store in a metadata table)
- Data drift (compare input data distribution to training data)
-
✅ Example Monitoring Table:
CREATE TABLE model_monitoring (
model_version STRING,
prediction_date TIMESTAMP,
metric_name STRING,
metric_value FLOAT
);
- ✅ Logging metrics:
from datetime import datetime
monitor_df = pd.DataFrame({
"model_version": ["v1.0"],
"prediction_date": [datetime.now()],
"metric_name": ["accuracy"],
"metric_value": [0.85]
})
session.create_dataframe(monitor_df).write.mode("append").save_as_table("MODEL_MONITORING")
4️⃣ Summary: Deployment & Monitoring
| Step | Tool/Method | Notes |
|---|---|---|
| Model training | Local Python / Notebook | Use sklearn, XGBoost, etc. |
| Model packaging | joblib / pickle | Save as .pkl or similar |
| Upload to Snowflake | Snowflake stage (PUT command) | Central location for model artifact |
| Inference | Snowpark Python script / stored proc | Load model, make predictions, save results |
| CI/CD automation | GitHub Actions / Azure DevOps | Automate updates and predictions |
| Monitoring | Snowflake tables for metrics | Track performance, drift, and model health |
5️⃣ Ready-to-use Example: Deploying & Monitoring Workflow
Here’s a minimal Snowpark-based workflow:
import joblib
import pandas as pd
from snowflake.snowpark import Session
def main(session: Session) -> str:
# Load data
df = session.table("CUSTOMER_DATA").to_pandas()
# Load trained model
session.file.get("@my_stage/my_model.pkl", "my_model.pkl")
model = joblib.load("my_model.pkl")
# Predict
predictions = model.predict(df[["AGE", "INCOME"]])
df["PREDICTION"] = predictions
# Save predictions
session.create_dataframe(df).write.mode("overwrite").save_as_table("PREDICTIONS")
# Monitoring
monitor_df = pd.DataFrame({
"model_version": ["v1.0"],
"prediction_date": [pd.Timestamp.now()],
"metric_name": ["accuracy"],
"metric_value": [0.85]
})
session.create_dataframe(monitor_df).write.mode("append").save_as_table("MODEL_MONITORING")
return "Predictions and monitoring updated!"
# If running locally:
if __name__ == "__main__":
from snowflake.snowpark import Session
connection_parameters = {
"account": "<account>",
"user": "<user>",
"password": "<password>",
"warehouse": "<warehouse>",
"database": "<database>",
"schema": "<schema>",
"role": "<role>"
}
session = Session.builder.configs(connection_parameters).create()
print(main(session))
- ✅ This script loads data, makes predictions, logs monitoring metrics, and saves everything to Snowflake!