Data Preprocessing and Feature Engineering
Raw data rarely arrives in a form suitable for machine learning. This module covers preprocessing techniques and feature engineering patterns using Snowpark and the Snowpark ML Preprocessing module.
Handling Nulls and Missing Values
Before training any model, address missing data. Snowpark provides DataFrame methods for null handling.
from snowflake.snowpark import Session
from snowflake.snowpark.functions import col, when, mean
df = session.table("RAW_CUSTOMERS")
# Drop rows where target column is null
df_clean = df.dropna(subset=["CHURN_FLAG"])
# Fill numeric nulls with column mean
avg_income = df.select(mean(col("INCOME"))).collect()[0][0]
df_clean = df_clean.fillna({"INCOME": avg_income, "TENURE_MONTHS": 0})
# Conditional fill
df_clean = df_clean.with_column(
"REGION",
when(col("REGION").is_null(), "UNKNOWN").otherwise(col("REGION"))
)
Type Casting and Normalization
Ensure columns have correct types before feeding them to models.
from snowflake.snowpark.types import FloatType, IntegerType
df = df.with_column("AGE", col("AGE").cast(IntegerType()))
df = df.with_column("BALANCE", col("BALANCE").cast(FloatType()))
Feature Engineering Patterns
One-Hot Encoding (Manual)
from snowflake.snowpark.functions import iff
categories = ["BASIC", "PREMIUM", "ENTERPRISE"]
for cat in categories:
df = df.with_column(f"PLAN_{cat}", iff(col("PLAN_TYPE") == cat, 1, 0))
Binning Continuous Variables
df = df.with_column(
"AGE_GROUP",
when(col("AGE") < 25, "YOUNG")
.when(col("AGE") < 45, "MIDDLE")
.when(col("AGE") < 65, "SENIOR")
.otherwise("ELDERLY")
)
Extracting Date Features
from snowflake.snowpark.functions import month, dayofweek, datediff, current_date
df = df.with_column("SIGNUP_MONTH", month(col("SIGNUP_DATE")))
df = df.with_column("SIGNUP_DOW", dayofweek(col("SIGNUP_DATE")))
df = df.with_column("DAYS_SINCE_SIGNUP", datediff("day", col("SIGNUP_DATE"), current_date()))
Train/Test Split
Split data before fitting any preprocessing transformers to avoid data leakage.
train_df, test_df = df.random_split([0.8, 0.2], seed=42)
Snowpark ML Preprocessing Module
The snowflake.ml.modeling.preprocessing module provides scikit-learn-compatible transformers that run natively in Snowflake.
| Transformer | Purpose | Key Parameters |
|---|---|---|
StandardScaler | Zero mean, unit variance | with_mean, with_std |
MinMaxScaler | Scale to a given range | feature_range |
OneHotEncoder | Categorical to binary columns | drop, handle_unknown |
OrdinalEncoder | Categorical to integer labels | categories |
LabelEncoder | Encode target labels | — |
StandardScaler Example
from snowflake.ml.modeling.preprocessing import StandardScaler
scaler = StandardScaler(
input_cols=["INCOME", "BALANCE", "TENURE_MONTHS"],
output_cols=["INCOME_SCALED", "BALANCE_SCALED", "TENURE_SCALED"]
)
scaler.fit(train_df)
train_scaled = scaler.transform(train_df)
test_scaled = scaler.transform(test_df)
OneHotEncoder Example
from snowflake.ml.modeling.preprocessing import OneHotEncoder
encoder = OneHotEncoder(
input_cols=["PLAN_TYPE", "REGION"],
output_cols=["PLAN_ENCODED", "REGION_ENCODED"],
drop="first",
handle_unknown="ignore"
)
encoder.fit(train_df)
train_encoded = encoder.transform(train_scaled)
Pipeline Pattern
Chain multiple preprocessing steps into a reproducible pipeline using sklearn.pipeline.Pipeline with Snowpark ML transformers.
from sklearn.pipeline import Pipeline
from snowflake.ml.modeling.preprocessing import StandardScaler, OrdinalEncoder
pipeline = Pipeline([
("ordinal", OrdinalEncoder(
input_cols=["REGION"],
output_cols=["REGION_ORD"]
)),
("scaler", StandardScaler(
input_cols=["INCOME", "BALANCE", "REGION_ORD"],
output_cols=["INCOME_S", "BALANCE_S", "REGION_S"]
)),
])
pipeline.fit(train_df)
train_ready = pipeline.transform(train_df)
test_ready = pipeline.transform(test_df)
Common Pitfalls
| Pitfall | Consequence | Remedy |
|---|---|---|
| Fitting on full dataset | Data leakage into test set | Always fit on train only |
| Ignoring null values | Model training errors | Handle nulls before scaling |
| Encoding high-cardinality columns | Dimension explosion | Use hashing or target encoding |
| Casting after scaling | Precision loss | Cast types before preprocessing |
Key Takeaways
- Clean nulls and cast types before any transformation.
- Always split into train/test before fitting transformers.
- Use Snowpark ML preprocessors for server-side execution — no data leaves Snowflake.
- Chain transformers in a pipeline for reproducibility.