Ask any working data scientist where their week actually goes, and it is rarely the fancy model. It is cleaning, filling, encoding, and scaling messy data, the part called Python ML preprocessing that quietly eats 60 to 80 percent of a real project. Get it wrong and your model can look brilliant in a notebook, then flop on live customers. This post covers handling missing values, encoding categories, scaling features, taming outliers, and wrapping it all in a leak-proof pipeline.

“Applied ML is basically feature engineering.”

Andrew Ng, Stanford CS229

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0, pandas 2.3.3 | Difficulty: Intermediate | Reading Time: 18 minutes

A data scientist named Pravin built a churn model that scored 95% in his notebook. He pushed it to production, popped the champagne, and watched it score 68% on real customers. What went wrong? He had fit the scaler on the whole dataset, test rows included, and he had quietly leaked the target into a feature he engineered. Both are preprocessing mistakes, and both are the kind that never throw an error. The model just lies to you in the notebook and then fails in front of your boss.

Raw data is messy. It has gaps where a sensor dropped out, text labels that algorithms cannot read, columns on wildly different scales, and the odd outlier that drags the whole model sideways. Before any algorithm sees your data, you have to clean it, convert it, and put every feature on a fair footing. Think of it like prepping ingredients before you cook. You wash, peel, and chop first, because no recipe works if you throw an unpeeled onion straight into the pan. This prep stage eats 60 to 80 percent of a real machine learning (ML) project, yet most tutorials skip it and hand you a tidy pre-cleaned CSV (comma-separated values) file.

Here is the catch. Each single step is easy. Filling one missing value is one line of code. The hard part is doing the steps in the right order, never letting test data sneak into training (that is data leakage), and wrapping it all in a pipeline you can replay on brand new data at prediction time. This post walks you through every step Pravin got wrong, so your notebook score and your production score finally agree.

Prerequisites

The diagram traces the journey raw data takes before it ever reaches a model: handle missing values (fill them in or drop them), encode categorical variables (one-hot or label encoding), scale numerical features (standardize or normalize), and deal with outliers, then split into train and test sets. Each stage uses a scikit-learn transformer that learns its settings from the training data and then applies them to both train and test. That fit-then-transform habit is what keeps the test set from leaking into training. The test rows must never get a vote on the preprocessing numbers.

Handling Missing Values

Picture a class survey where some students left a few boxes blank. You would not shred every half-filled form; you would either fill the blanks with a sensible guess or set a handful aside. That is exactly the choice you face with missing data. Most ML algorithms choke on a missing value. Feed scikit-learn a NaN and it raises an error rather than guess. You have three ways out: drop the rows (you lose data), fill the gap with a statistic like the mean, median, or most common value, or predict the missing value from the other columns.

Which one you pick depends on how much is missing and whether the gaps are random or follow a pattern. A quick gut check: if a column is 60 percent empty, no clever fill will save it, drop the column. If it is 5 percent empty, filling is almost always the right call.

📄 missing_values.py: see where the gaps are first

import pandas as pd
import numpy as np

# A small employee table with gaps in every column
df = pd.DataFrame({
    "age": [25, 30, np.nan, 28, 35, np.nan, 32, 27, 29, 31],
    "salary": [50000, np.nan, 72000, 45000, 90000, 55000, np.nan, 48000, 62000, 70000],
    "department": ["Engineering", "Sales", "Engineering", None, "Sales",
                   "Marketing", "Engineering", "Sales", None, "Marketing"],
    "performance": [8.5, 7.2, 9.1, 6.8, np.nan, 7.5, 8.0, np.nan, 7.8, 8.3]
})

print("Original data:")
print(df)
print(f"\nMissing values per column:\n{df.isnull().sum()}")
print(f"\nMissing percentage per column:")
print((df.isnull().mean() * 100).round(1))

▶ Output

Original data:
    age   salary   department  performance
0  25.0  50000.0  Engineering          8.5
1  30.0      NaN        Sales          7.2
2   NaN  72000.0  Engineering          9.1
3  28.0  45000.0         None          6.8
4  35.0  90000.0        Sales          NaN
5   NaN  55000.0    Marketing          7.5
6  32.0      NaN  Engineering          8.0
7  27.0  48000.0        Sales          NaN
8  29.0  62000.0         None          7.8
9  31.0  70000.0    Marketing          8.3

Missing values per column:
age            2
salary         2
department     2
performance    2
dtype: int64

Missing percentage per column:
age            20.0
salary         20.0
department     20.0
performance    20.0
dtype: float64

📄 imputation_strategies.py: four ways to fill a gap

import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer

# Numerical features: compare strategies on the same ages
ages = np.array([25, 30, np.nan, 28, 35, np.nan, 32, 27, 29, 31]).reshape(-1, 1)

# Strategy 1: Mean imputation
mean_imp = SimpleImputer(strategy="mean")
ages_mean = mean_imp.fit_transform(ages)
print(f"Mean imputation: missing becomes {ages_mean[2][0]:.1f} (mean of others)")

# Strategy 2: Median imputation (robust to outliers)
median_imp = SimpleImputer(strategy="median")
ages_median = median_imp.fit_transform(ages)
print(f"Median imputation: missing becomes {ages_median[2][0]:.1f}")

# Strategy 3: KNN imputation (borrows from similar rows)
knn_imp = KNNImputer(n_neighbors=3)
data = np.array([
    [25, 50000], [30, 65000], [np.nan, 72000],
    [28, 45000], [35, 90000], [np.nan, 55000]
])
data_knn = knn_imp.fit_transform(data)
print(f"KNN imputation: missing ages become {data_knn[2, 0]:.1f}, {data_knn[5, 0]:.1f}")
print("  (KNN uses salary similarity to estimate age)")

# Categorical features: use mode (most frequent). Mark gaps with np.nan
# so SimpleImputer treats them as missing, not as a real category.
cat_imp = SimpleImputer(strategy="most_frequent")
depts = np.array(["Engineering", "Sales", "Engineering", np.nan,
                  "Sales", "Marketing", "Engineering", "Sales", np.nan, "Marketing"],
                 dtype=object).reshape(-1, 1)
depts_filled = cat_imp.fit_transform(depts)
print(f"\nCategorical mode imputation: missing becomes '{depts_filled[3][0]}'")

▶ Output

Mean imputation: missing becomes 29.6 (mean of others)
Median imputation: missing becomes 29.5
KNN imputation: missing ages become 30.0, 27.7
  (KNN uses salary similarity to estimate age)

Categorical mode imputation: missing becomes 'Engineering'

What happened here: Mean and median paste the same single number into every gap, so two very different people can end up with the same imputed age. KNN (K-Nearest Neighbors) is smarter. It looks at the rows nearest to each gap and borrows from them, the way you might guess a stranger’s age from the friends they hang out with. Using salary as the clue, it figured the person earning 72,000 is around 30.0 and the person earning 55,000 is around 27.7. For text columns, mode fills with the most common category, here ‘Engineering’.

Rule of thumb: median for numbers with outliers, KNN when the columns move together, mode for categories. One catch with categories: tag the gaps with np.nan, not None, or SimpleImputer will treat None as a real label and happily fill the most frequent “missing”.

Encoding Categorical Variables

ML algorithms do math, and you cannot multiply the word “Sales”. So every text category has to become a number. The lazy way is to number them 0, 1, 2, but that quietly tells the model that Marketing (1) is bigger than Engineering (0), which is nonsense. Picture giving each department a jersey number and then a coach assuming the player wearing 2 is twice as good as the one wearing 1. One-hot encoding fixes this by giving each category its own yes-or-no column, so no category looks bigger than another.

📄 encoding.py: label vs one-hot vs ordinal

import numpy as np
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, OrdinalEncoder

departments = ["Engineering", "Sales", "Marketing", "Engineering", "Sales"]

# Label Encoding: assigns one integer per category
le = LabelEncoder()
label_encoded = le.fit_transform(departments)
mapping = {str(cls): int(code) for cls, code in zip(le.classes_, le.transform(le.classes_))}
print(f"Label Encoding: {mapping}")
print(f"  Result: {label_encoded}")
print(f"  Problem: model thinks Marketing(1) > Engineering(0)\n")

# One-Hot Encoding: one binary column per category
ohe = OneHotEncoder(sparse_output=False)
onehot = ohe.fit_transform(np.array(departments).reshape(-1, 1))
print(f"One-Hot Encoding:")
print(f"  Categories: {ohe.categories_[0]}")
for dept, row in zip(departments, onehot):
    print(f"  {dept:<12} becomes {row.astype(int)}")

# Ordinal Encoding: ONLY when the order is real
education = ["High School", "Bachelor", "Master", "PhD", "Bachelor"]
oe = OrdinalEncoder(categories=[["High School", "Bachelor", "Master", "PhD"]])
ordinal = oe.fit_transform(np.array(education).reshape(-1, 1))
print(f"\nOrdinal Encoding (education has a natural order):")
for edu, val in zip(education, ordinal.ravel()):
    print(f"  {edu:<12} becomes {val:.0f}")

▶ Output

Label Encoding: {'Engineering': 0, 'Marketing': 1, 'Sales': 2}
  Result: [0 2 1 0 2]
  Problem: model thinks Marketing(1) > Engineering(0)

One-Hot Encoding:
  Categories: ['Engineering' 'Marketing' 'Sales']
  Engineering  becomes [1 0 0]
  Sales        becomes [0 0 1]
  Marketing    becomes [0 1 0]
  Engineering  becomes [1 0 0]
  Sales        becomes [0 0 1]

Ordinal Encoding (education has a natural order):
  High School  becomes 0
  Bachelor     becomes 1
  Master       becomes 2
  PhD          becomes 3
  Bachelor     becomes 1

What happened here: Label encoding is fine for the target you are predicting (y), or for tree-based models that just split on a threshold and do not care about order. But for linear models, Support Vector Machines (SVMs), and KNN, reach for one-hot, so you never imply an order that does not exist. Save ordinal encoding for columns that genuinely have a ladder built in, like education level (High School, Bachelor, Master, PhD) or T-shirt size (S, M, L, XL).

One more reason to notice the difference in the output: label encoding kept your data in a single column, while one-hot grew it into three. That column growth is the trade-off, and it comes back to bite you later in the high-cardinality mistake.

Feature Scaling

Features on different scales quietly bully distance-based and gradient-based algorithms. Say salary runs from 30,000 to 200,000 and age runs from 20 to 60. When the algorithm measures how far apart two people are, the salary numbers are so much bigger that age barely registers, and the model basically ignores it. It is like comparing two runners where one time is in seconds and the other in milliseconds. Unless you convert to the same unit, one number drowns out the other. Scaling rewrites every feature onto a comparable range so each one gets a fair say.

📄 scaling.py: StandardScaler vs MinMaxScaler vs RobustScaler

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Data with different scales + an outlier
data = np.array([
    [25, 50000, 3],
    [30, 65000, 5],
    [28, 55000, 4],
    [32, 70000, 7],
    [60, 200000, 2],  # Outlier: senior executive
]).astype(float)

features = ["Age", "Salary", "Experience"]

scalers = {
    "StandardScaler (mean=0, std=1)": StandardScaler(),
    "MinMaxScaler (0 to 1)": MinMaxScaler(),
    "RobustScaler (uses median/IQR)": RobustScaler(),
}

for name, scaler in scalers.items():
    scaled = scaler.fit_transform(data)
    print(f"\n{name}:")
    print(f"  {'Feature':<12} {'Min':>8} {'Max':>8} {'Mean':>8}")
    for i, feat in enumerate(features):
        col = scaled[:, i]
        print(f"  {feat:<12} {col.min():>8.2f} {col.max():>8.2f} {col.mean():>8.2f}")

▶ Output

StandardScaler (mean=0, std=1):
  Feature           Min      Max     Mean
  Age             -0.79     1.97     0.00
  Salary          -0.67     1.98     0.00
  Experience      -1.28     1.63    -0.00

MinMaxScaler (0 to 1):
  Feature           Min      Max     Mean
  Age              0.00     1.00     0.29
  Salary           0.00     1.00     0.25
  Experience       0.00     1.00     0.44

RobustScaler (uses median/IQR):
  Feature           Min      Max     Mean
  Age             -1.25     7.50     1.25
  Salary          -1.00     9.00     1.53
  Experience      -1.00     1.50     0.10

What happened here: StandardScaler recenters every feature to a mean of 0 and a standard deviation of 1. That is the safe default and the one you will reach for most. MinMaxScaler squashes everything into the range 0 to 1, which is handy when you already know the natural minimum and maximum. RobustScaler uses the median and the interquartile range instead of the mean and standard deviation, so that one senior executive earning 200,000 does not yank the scale around.

Notice how the executive row blew the RobustScaler salary max up to 9.00 while the StandardScaler max stayed near 1.98. When your data has loud outliers, RobustScaler keeps them from shouting over everyone else. The rule that ties it all together: fit the scaler on training data only, then transform both train and test with that same fitted scaler.

Preprocessing Pipeline: Do It Right, Once

Doing all of this by hand is tedious, and the day you forget one step or fit a scaler on the wrong rows is the day a bug slips into production. Scikit-learn's Pipeline and ColumnTransformer bundle every Python ML preprocessing step plus the model into one object. Think of it as a recipe card you write once: impute, scale, encode, then fit the model, always in that order. You hand the card the raw data and it runs the whole sequence the same way every time. Fit it once on the training data, and it replays the exact same transformations on new data at prediction time. No leakage, no forgotten steps, no surprises.

📄 pipeline.py: one object, every step, zero leakage

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)

# Create realistic dataset
n = 200
df = pd.DataFrame({
    "age": rng.integers(22, 55, n).astype(float),
    "salary": rng.integers(30000, 150000, n).astype(float),
    "department": rng.choice(["Engineering", "Sales", "Marketing", "HR"], n),
    "experience": rng.integers(1, 20, n).astype(float),
    "promoted": rng.integers(0, 2, n)
})

# Inject missing values
df.loc[rng.choice(n, 15, replace=False), "age"] = np.nan
df.loc[rng.choice(n, 10, replace=False), "salary"] = np.nan
df.loc[rng.choice(n, 8, replace=False), "department"] = None

X = df.drop("promoted", axis=1)
y = df["promoted"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define preprocessing for each column type
numeric_features = ["age", "salary", "experience"]
categorical_features = ["department"]

numeric_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer([
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
])

# Full pipeline: preprocessing + model
full_pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(random_state=42))
])

# One call does everything
full_pipeline.fit(X_train, y_train)
test_acc = full_pipeline.score(X_test, y_test)

print(f"Pipeline steps: {[step[0] for step in full_pipeline.steps]}")
print(f"Test accuracy: {test_acc:.1%}")
print(f"\nPredict on new data (handles missing values automatically):")
new_employee = pd.DataFrame({
    "age": [28], "salary": [None], "department": ["Engineering"], "experience": [3]
})
prob = full_pipeline.predict_proba(new_employee)[0]
print(f"  Anvi (28, Engineering, 3yr): {prob[1]:.0%} promotion probability")

▶ Output

Pipeline steps: ['preprocessor', 'classifier']
Test accuracy: 37.5%

Predict on new data (handles missing values automatically):
  Anvi (28, Engineering, 3yr): 40% promotion probability

What happened here: The pipeline does the whole chore for you: fill missing ages with the median, fill missing departments with the mode, scale the numeric columns, one-hot encode the department, then feed the clean result to logistic regression. One fit() call trains all of it on the training rows only. One predict_proba() call preprocesses and scores brand new data, including patching new applicant Anvi's missing salary on the fly, with no extra code from you.

Do not read anything into the 37.5 percent accuracy. We generated the "promoted" labels with a coin flip, so there is no real pattern to learn, and on a small 40-row test set a score bouncing around 50 percent either way is exactly what chance looks like. The point of this example is the plumbing, not the score.

Common Mistakes

❌ Mistake 1: Fitting scaler on entire dataset (data leakage)

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
import numpy as np

X = np.random.default_rng(42).normal(size=(100, 3))
y = (X[:, 0] > 0).astype(int)

# BAD: Fit scaler on ALL data before splitting
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(X)  # Test info leaks!
X_train, X_test = X_all_scaled[:80], X_all_scaled[80:]

# GOOD: Split first, fit on train only
X_train_raw, X_test_raw = X[:80], X[80:]
scaler = StandardScaler()
X_train_good = scaler.fit_transform(X_train_raw)
X_test_good = scaler.transform(X_test_raw)  # transform, NOT fit_transform

# BEST: Use a Pipeline, it handles this automatically
print("Always split BEFORE preprocessing")
print("Use Pipeline to make data leakage impossible")

Why this bites: in the BAD block, fit_transform(X) computes the mean and standard deviation from every row, test rows included. The scaler now knows something about the test set before training ever happens, so your test score is flattering and fake. This is exactly the bug that dropped Pravin from 95 percent to 68 percent. The GOOD block fixes it by fitting on the training rows only and calling transform (not fit_transform) on the test rows. The BEST fix is to never split this logic by hand at all: put the scaler in a Pipeline and let scikit-learn guarantee the test set stays sealed.

❌ Mistake 2: One-hot encoding high-cardinality features

import numpy as np

# If a column has 500 unique cities, one-hot creates 500 new columns
# This causes: slow training, overfitting, memory issues

# Solutions for high-cardinality categoricals:
# 1. Target encoding (replace category with target mean)
# 2. Frequency encoding (replace with count)
# 3. Group rare categories into "Other"
# 4. Use tree-based models that handle categories natively

cities = ["Mumbai", "Delhi", "Pune"] * 30 + ["Nagpur", "Jaipur"] * 5
unique = len(set(cities))
print(f"Cities: {unique} unique, so one-hot creates {unique} columns")
print("For 500 cities, that's 500 sparse columns. Use target encoding instead.")

▶ Output

Cities: 5 unique, so one-hot creates 5 columns
For 500 cities, that's 500 sparse columns. Use target encoding instead.

Why this bites: one-hot encoding adds one new column per category. Five cities cost you five columns, no big deal. But a "city" column with 500 unique values explodes into 500 mostly-empty columns, which slows training to a crawl, eats memory, and hands the model a perfect way to memorize noise (overfitting). When a category has lots of distinct values (a high-cardinality column), reach for target encoding, frequency encoding, or simply bucket the rare ones into an "Other" group. Tree-based models like random forests and gradient boosting can also swallow many categories without the column blow-up.

Practice Exercises

1. Exercise 1: Take the employee table from the first example and build a ColumnTransformer that imputes the numeric columns with the median and one-hot encodes the department column. Print the shape before and after to see how many columns one-hot added.
2. Exercise 2: Re-run the scaling example, but first add a second outlier row (say a 70-year-old earning 500,000). Compare how StandardScaler and RobustScaler each react. Which one barely moves?
3. Exercise 3: Deliberately reproduce the data leakage bug: fit a scaler on the full dataset, then fit it the correct way on the training split only. Print both sets of means and standard deviations and confirm they differ.

Wrapping Up

You now have the full Python ML preprocessing playbook: spot and fill missing values, encode categories without inventing a fake order, scale features so no single column bullies the rest, tame outliers, and wrap the whole thing in a Pipeline that makes data leakage nearly impossible. That last habit, fit on the training rows only and let the pipeline replay the exact same transformations at prediction time, is what keeps your notebook score and your production score honest. It is the one fix that would have saved Pravin from his 95-to-68 percent faceplant.

Next up is feature engineering, where you stop merely cleaning data and start creating brand new signal from it: ratios, date parts, and interaction terms that hand your model patterns it could never find on its own. Ready for the whole journey from beginner to job-ready? Explore the full Python + AI/ML tutorial series home and work through it in order.

Frequently Asked Questions

Interview Questions on Python ML Preprocessing

Scenario questions, not trivia: this is the form this topic takes in a real interview.

Q: What is the difference between fit, transform, and fit_transform on a scikit-learn transformer?

fit learns the parameters from data (for example the mean and standard deviation a scaler needs) and stores them. transform applies those stored parameters to data. fit_transform does both in one step and should only ever touch the training set. On the test set you call transform alone, so the test rows are scaled with training statistics and never influence them.

Q: When would you pick median imputation over mean imputation?

Pick the median when the column is skewed or has outliers, like salary or house price. The mean gets dragged toward extreme values, so a couple of very high earners would inflate every filled-in gap. The median sits at the middle of the sorted values and barely moves when a few points are extreme, which makes it the safer default for real-world numeric columns.

Q: You one-hot encode a user_id column with 50,000 unique values, and training slows to a crawl while memory usage explodes. What do you check first and how do you fix it?

The first thing to check is the cardinality of every categorical column, because one-hot encoding creates one column per unique value, so 50,000 IDs become 50,000 sparse columns. An identifier like user_id usually should not be a model feature at all. If you do need the signal, switch to target encoding or frequency encoding, or bucket rare values into an "Other" group. Tree-based models such as random forests and gradient boosting can also absorb high-cardinality categories without the column blow-up.

Q: Your pipeline applies StandardScaler to columns that were just one-hot encoded into 0/1 values. Is that a problem, and how do you structure it correctly?

Scaling already-binary one-hot columns is usually pointless and can make coefficients harder to read, though it rarely breaks a model outright. The clean structure is a ColumnTransformer that routes numeric columns to an imputer plus scaler and categorical columns to an imputer plus one-hot encoder, so each column type gets only the steps it needs. That way the scaler never sees the encoded columns, and the whole thing still lives inside one leak-proof pipeline.

Q: How does KNNImputer decide what value to fill in, and when is it worth the extra cost over SimpleImputer?

KNNImputer finds the rows most similar to the one with the gap, using the other columns as clues, then fills the gap with a distance-weighted value from those neighbors. It shines when features move together, for example estimating a missing age from salary and experience. It costs more compute than SimpleImputer because it measures distances across the dataset, so reserve it for correlated numeric data rather than a quick one-line mean fill.

Q: Why does wrapping preprocessing inside a Pipeline matter specifically during cross-validation?

During cross-validation the data is split into folds many times over. If you scale or impute once up front, every fold's "training" portion has already been contaminated by statistics from its "validation" portion, which quietly inflates your scores. A Pipeline re-fits every preprocessing step on each fold's training slice only, so validation data stays sealed on every single split and your cross-validation numbers stay trustworthy.

Series: Python + AI/ML Cookbook, Part 5: Machine Learning

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: ML: Train/Test Split & Cross-Validation

Next: ML: Feature Engineering, Turning Raw Data into Predictive Power

Series Home: Python + AI/ML Tutorial Series