🚀 Using Snowpark ML & Integrations with scikit-learn, XGBoost
📌 Overview
Snowflake’s Snowpark ML offers seamless integration with popular Python ML libraries like scikit-learn and XGBoost. You can:
- ✅ Load data from Snowflake
- ✅ Train models using
scikit-learnorXGBoost(outside or inside Snowflake) - ✅ Deploy trained models inside Snowflake for scoring
- ✅ Monitor & iterate easily
1️⃣ Setup & Libraries
First, ensure you have these libraries:
pip install snowflake-snowpark-python scikit-learn xgboost pandas joblib
2️⃣ Using scikit-learn or XGBoost for Model Training
⚡ Example with scikit-learn
import pandas as pd
from snowflake.snowpark import Session
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Connect to Snowflake
connection_parameters = {
"account": "<account>",
"user": "<user>",
"password": "<password>",
"warehouse": "<warehouse>",
"database": "<database>",
"schema": "<schema>"
}
session = Session.builder.configs(connection_parameters).create()
# Load data from Snowflake
df = session.table("CUSTOMER_DATA").to_pandas()
# Preprocess data
X = df[["feature1", "feature2"]]
y = df["label"]
# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Train scikit-learn model
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)
# Evaluate
y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")
⚡ Example with XGBoost
import xgboost as xgb
# Prepare data
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
# Train XGBoost model
params = {"objective": "binary:logistic", "eval_metric": "logloss"}
model = xgb.train(params, dtrain, num_boost_round=100)
# Predict
y_pred_prob = model.predict(dtest)
y_pred = (y_pred_prob > 0.5).astype(int)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")
3️⃣ Model Serialization & Deployment
✅ Save Model
import joblib
joblib.dump(model, "model.pkl")
✅ Upload to Snowflake Stage
PUT file://model.pkl @my_stage;
4️⃣ Inference Using Snowpark (Inside Snowflake)
Create a Snowpark stored procedure that loads the trained model and applies it:
import joblib
from snowflake.snowpark import Session
def score_in_snowflake(session: Session) -> str:
# Load data
df = session.table("CUSTOMER_DATA").to_pandas()
# Load model from stage
session.file.get("@my_stage/model.pkl", "model.pkl")
model = joblib.load("model.pkl")
# Predict
features = df[["feature1", "feature2"]]
predictions = model.predict(features)
df["PREDICTION"] = predictions
# Save predictions back to Snowflake
session.create_dataframe(df).write.mode("overwrite").save_as_table("PREDICTIONS")
return "Predictions stored!"
# Local test
if __name__ == "__main__":
session = Session.builder.configs(connection_parameters).create()
print(score_in_snowflake(session))
5️⃣ Monitoring & Iteration
- ✅ Save model metrics (e.g., accuracy, precision) in a MODEL_METRICS table.
- ✅ Schedule retraining as needed (e.g., via CI/CD pipeline or Airflow).
- ✅ Keep models versioned for reproducibility.
6️⃣ Summary Table
| Aspect | Tools / Libraries | Notes |
|---|---|---|
| Data Processing | Snowpark, pandas | Use Snowpark for direct Snowflake data access |
| Model Training | scikit-learn, XGBoost | Done outside Snowflake, optionally using Snowpark ML APIs |
| Model Inference | Snowpark (stored procedures) | Deploy models inside Snowflake for scoring |
| Monitoring | Snowflake tables (MODEL_METRICS) | Save metrics for drift & performance tracking |
| Automation | GitHub Actions, Airflow, or other CI/CD | Deploy + monitor pipelines automatically |