Skip to main content

Data Preprocessing and Feature Engineering

Raw data rarely arrives in a form suitable for machine learning. This module covers preprocessing techniques and feature engineering patterns using Snowpark and the Snowpark ML Preprocessing module.

Handling Nulls and Missing Values​

Before training any model, address missing data. Snowpark provides DataFrame methods for null handling.

from snowflake.snowpark import Session
from snowflake.snowpark.functions import col, when, mean

df = session.table("RAW_CUSTOMERS")

# Drop rows where target column is null
df_clean = df.dropna(subset=["CHURN_FLAG"])

# Fill numeric nulls with column mean
avg_income = df.select(mean(col("INCOME"))).collect()[0][0]
df_clean = df_clean.fillna({"INCOME": avg_income, "TENURE_MONTHS": 0})

# Conditional fill
df_clean = df_clean.with_column(
"REGION",
when(col("REGION").is_null(), "UNKNOWN").otherwise(col("REGION"))
)

Type Casting and Normalization​

Ensure columns have correct types before feeding them to models.

from snowflake.snowpark.types import FloatType, IntegerType

df = df.with_column("AGE", col("AGE").cast(IntegerType()))
df = df.with_column("BALANCE", col("BALANCE").cast(FloatType()))

Feature Engineering Patterns​

One-Hot Encoding (Manual)​

from snowflake.snowpark.functions import iff

categories = ["BASIC", "PREMIUM", "ENTERPRISE"]
for cat in categories:
df = df.with_column(f"PLAN_{cat}", iff(col("PLAN_TYPE") == cat, 1, 0))

Binning Continuous Variables​

df = df.with_column(
"AGE_GROUP",
when(col("AGE") < 25, "YOUNG")
.when(col("AGE") < 45, "MIDDLE")
.when(col("AGE") < 65, "SENIOR")
.otherwise("ELDERLY")
)

Extracting Date Features​

from snowflake.snowpark.functions import month, dayofweek, datediff, current_date

df = df.with_column("SIGNUP_MONTH", month(col("SIGNUP_DATE")))
df = df.with_column("SIGNUP_DOW", dayofweek(col("SIGNUP_DATE")))
df = df.with_column("DAYS_SINCE_SIGNUP", datediff("day", col("SIGNUP_DATE"), current_date()))

Train/Test Split​

Split data before fitting any preprocessing transformers to avoid data leakage.

train_df, test_df = df.random_split([0.8, 0.2], seed=42)

Snowpark ML Preprocessing Module​

The snowflake.ml.modeling.preprocessing module provides scikit-learn-compatible transformers that run natively in Snowflake.

TransformerPurposeKey Parameters
StandardScalerZero mean, unit variancewith_mean, with_std
MinMaxScalerScale to a given rangefeature_range
OneHotEncoderCategorical to binary columnsdrop, handle_unknown
OrdinalEncoderCategorical to integer labelscategories
LabelEncoderEncode target labels—

StandardScaler Example​

from snowflake.ml.modeling.preprocessing import StandardScaler

scaler = StandardScaler(
input_cols=["INCOME", "BALANCE", "TENURE_MONTHS"],
output_cols=["INCOME_SCALED", "BALANCE_SCALED", "TENURE_SCALED"]
)

scaler.fit(train_df)
train_scaled = scaler.transform(train_df)
test_scaled = scaler.transform(test_df)

OneHotEncoder Example​

from snowflake.ml.modeling.preprocessing import OneHotEncoder

encoder = OneHotEncoder(
input_cols=["PLAN_TYPE", "REGION"],
output_cols=["PLAN_ENCODED", "REGION_ENCODED"],
drop="first",
handle_unknown="ignore"
)

encoder.fit(train_df)
train_encoded = encoder.transform(train_scaled)

Pipeline Pattern​

Chain multiple preprocessing steps into a reproducible pipeline using sklearn.pipeline.Pipeline with Snowpark ML transformers.

from sklearn.pipeline import Pipeline
from snowflake.ml.modeling.preprocessing import StandardScaler, OrdinalEncoder

pipeline = Pipeline([
("ordinal", OrdinalEncoder(
input_cols=["REGION"],
output_cols=["REGION_ORD"]
)),
("scaler", StandardScaler(
input_cols=["INCOME", "BALANCE", "REGION_ORD"],
output_cols=["INCOME_S", "BALANCE_S", "REGION_S"]
)),
])

pipeline.fit(train_df)
train_ready = pipeline.transform(train_df)
test_ready = pipeline.transform(test_df)

Common Pitfalls​

PitfallConsequenceRemedy
Fitting on full datasetData leakage into test setAlways fit on train only
Ignoring null valuesModel training errorsHandle nulls before scaling
Encoding high-cardinality columnsDimension explosionUse hashing or target encoding
Casting after scalingPrecision lossCast types before preprocessing

Key Takeaways​

  • Clean nulls and cast types before any transformation.
  • Always split into train/test before fitting transformers.
  • Use Snowpark ML preprocessors for server-side execution — no data leaves Snowflake.
  • Chain transformers in a pipeline for reproducibility.