Skip to main content

🚀 Using Snowpark ML & Integrations with scikit-learn, XGBoost

📌 Overview​

Snowflake’s Snowpark ML offers seamless integration with popular Python ML libraries like scikit-learn and XGBoost. You can:

  • ✅ Load data from Snowflake
  • ✅ Train models using scikit-learn or XGBoost (outside or inside Snowflake)
  • ✅ Deploy trained models inside Snowflake for scoring
  • ✅ Monitor & iterate easily

1️⃣ Setup & Libraries​

First, ensure you have these libraries:

pip install snowflake-snowpark-python scikit-learn xgboost pandas joblib

2️⃣ Using scikit-learn or XGBoost for Model Training​

⚡ Example with scikit-learn​

import pandas as pd
from snowflake.snowpark import Session
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Connect to Snowflake
connection_parameters = {
"account": "<account>",
"user": "<user>",
"password": "<password>",
"warehouse": "<warehouse>",
"database": "<database>",
"schema": "<schema>"
}
session = Session.builder.configs(connection_parameters).create()

# Load data from Snowflake
df = session.table("CUSTOMER_DATA").to_pandas()

# Preprocess data
X = df[["feature1", "feature2"]]
y = df["label"]

# Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train scikit-learn model
model = RandomForestClassifier(n_estimators=100)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

⚡ Example with XGBoost​

import xgboost as xgb

# Prepare data
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

# Train XGBoost model
params = {"objective": "binary:logistic", "eval_metric": "logloss"}
model = xgb.train(params, dtrain, num_boost_round=100)

# Predict
y_pred_prob = model.predict(dtest)
y_pred = (y_pred_prob > 0.5).astype(int)

print(f"Accuracy: {accuracy_score(y_test, y_pred)}")

3️⃣ Model Serialization & Deployment​

✅ Save Model​

import joblib

joblib.dump(model, "model.pkl")

✅ Upload to Snowflake Stage​

PUT file://model.pkl @my_stage;

4️⃣ Inference Using Snowpark (Inside Snowflake)​

Create a Snowpark stored procedure that loads the trained model and applies it:

import joblib
from snowflake.snowpark import Session

def score_in_snowflake(session: Session) -> str:
# Load data
df = session.table("CUSTOMER_DATA").to_pandas()

# Load model from stage
session.file.get("@my_stage/model.pkl", "model.pkl")
model = joblib.load("model.pkl")

# Predict
features = df[["feature1", "feature2"]]
predictions = model.predict(features)
df["PREDICTION"] = predictions

# Save predictions back to Snowflake
session.create_dataframe(df).write.mode("overwrite").save_as_table("PREDICTIONS")

return "Predictions stored!"

# Local test
if __name__ == "__main__":
session = Session.builder.configs(connection_parameters).create()
print(score_in_snowflake(session))

5️⃣ Monitoring & Iteration​

  • ✅ Save model metrics (e.g., accuracy, precision) in a MODEL_METRICS table.
  • ✅ Schedule retraining as needed (e.g., via CI/CD pipeline or Airflow).
  • ✅ Keep models versioned for reproducibility.

6️⃣ Summary Table​

AspectTools / LibrariesNotes
Data ProcessingSnowpark, pandasUse Snowpark for direct Snowflake data access
Model Trainingscikit-learn, XGBoostDone outside Snowflake, optionally using Snowpark ML APIs
Model InferenceSnowpark (stored procedures)Deploy models inside Snowflake for scoring
MonitoringSnowflake tables (MODEL_METRICS)Save metrics for drift & performance tracking
AutomationGitHub Actions, Airflow, or other CI/CDDeploy + monitor pipelines automatically