You just inherited a dataset with 100 columns and a deadline, and Python feature selection is the calmer answer to both. Instead of feeding every column to your model and hoping for the best, it works out which ones actually matter and quietly drops the rest, so training runs faster and overfits less. Here is how to pick the right method without guessing, using real data at every step.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

Antoine de Saint-Exupéry

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Intermediate | Reading Time: 17 minutes

Think about packing a bag for a weekend trip. You could throw in everything you own, but then the bag is heavy, you can never find your toothbrush, and half the stuff never gets used. So you pick the few things you actually need. Feature selection in Python does the same job for your data: out of 50, 100, or even 500 columns, it keeps the ones that help the model and tosses the rest.

Not every column earns its place. Some are simply irrelevant (your shoe size does not predict your income). Some are redundant (height in centimetres and height in inches say the exact same thing twice). Some are just noise that confuses the model. Feature selection spots the columns worth keeping and drops the dead weight.

Less is often more. A model built on 15 well-chosen features usually beats a model built on 100 features where half are junk. Fewer features mean faster training, less overfitting, simpler explanations, and a cheaper model to run in production.

There are three families of methods, and the right one depends on how big your dataset is, how many features you have, and how much time you can spend. This post walks through all three, then hands you a decision framework so you can pick in about 30 seconds.

Prerequisites

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The flowchart guides you through three feature selection approaches. Filter methods (correlation, variance threshold) are fast but ignore how features work together. Wrapper methods (forward and backward selection) are thorough but slow because they train a model over and over. Embedded methods (L1 logistic regression, tree-based importance) are built right into model training, so you get selection almost for free. The first fork is simply dataset size: with a huge dataset, filter methods give you a quick first pass, and with a smaller one, wrapper methods become practical. The code examples below run all three on the same dataset so you can see them side by side.

Python Feature Selection Method Comparison

Before the code, here is the whole landscape on one screen. Skim this table, find the row that matches your situation, and you already know which family to reach for.

Filter Methods: Fast Screening

Filter methods score each feature on its own using simple statistics. No model gets trained, which makes them blazing fast. Think of it like a bouncer at a club checking IDs one by one: quick, but he never sees how two guests behave together. That is the trade-off here. Filter methods cannot spot a feature that only matters when paired with another feature. Still, as a first pass at Python feature selection, nothing beats their speed.

📄 filter_methods.py: variance threshold, correlation, mutual information

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.feature_selection import VarianceThreshold, mutual_info_classif
from sklearn.feature_selection import SelectKBest, f_classif

# Generate dataset: 10 informative features + 10 noise features
X, y = make_classification(n_samples=500, n_features=20, n_informative=10,
                            n_redundant=3, n_clusters_per_class=1, random_state=42)
feature_names = [f"feat_{i}" for i in range(20)]

# 1. Variance Threshold: remove near-constant features
# Add a constant feature to demonstrate
X_with_const = np.column_stack([X, np.ones(500)])
vt = VarianceThreshold(threshold=0.01)
X_filtered = vt.fit_transform(X_with_const)
print(f"Variance Threshold: {X_with_const.shape[1]} -> {X_filtered.shape[1]} features")
print(f"  Removed {X_with_const.shape[1] - X_filtered.shape[1]} near-constant features")

# 2. Correlation-based: remove highly correlated features
# Add a near-duplicate of feat_0 (same info in different units, plus tiny noise)
df = pd.DataFrame(X, columns=feature_names)
df["feat_0_dup"] = df["feat_0"] * 2.5 + np.random.default_rng(0).normal(0, 0.01, 500)
corr_matrix = df.corr().abs()
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
high_corr = [col for col in upper.columns if any(upper[col] > 0.85)]
print(f"\nCorrelation filter (threshold=0.85): remove {len(high_corr)} redundant feature(s)")
print(f"  Columns to drop: {high_corr}")

# 3. Mutual Information: measures dependency (works for non-linear too)
mi_scores = mutual_info_classif(X, y, random_state=42)
mi_ranking = pd.Series(mi_scores, index=feature_names).sort_values(ascending=False)
print(f"\nMutual Information (top 5):")
for feat, score in mi_ranking.head(5).items():
    print(f"  {feat}: {score:.4f}")

# 4. ANOVA F-test: for classification
selector = SelectKBest(f_classif, k=10)
X_selected = selector.fit_transform(X, y)
selected = np.array(feature_names)[selector.get_support()]
print(f"\nANOVA F-test (top 10): {selected.tolist()}")

▶ Output

Variance Threshold: 21 -> 20 features
  Removed 1 near-constant features

Correlation filter (threshold=0.85): remove 1 redundant feature(s)
  Columns to drop: ['feat_0_dup']

Mutual Information (top 5):
  feat_7: 0.2476
  feat_11: 0.2081
  feat_18: 0.1629
  feat_19: 0.1474
  feat_16: 0.1352

ANOVA F-test (top 10): ['feat_3', 'feat_6', 'feat_7', 'feat_11', 'feat_12', 'feat_13', 'feat_16', 'feat_17', 'feat_18', 'feat_19']

What happened here: Four quick filters, four different angles. Variance threshold caught the one column that never changes and dropped it. The correlation filter found the duplicate we planted (feat_0_dup is just feat_0 in different units) and flagged it for removal. Mutual information ranked features by how much each one tells us about the target, and the ANOVA (Analysis of Variance) F-test picked its top 10. Notice that the methods do not agree on a single winner. That is normal and expected. Each filter measures a slightly different kind of usefulness, so you usually run a couple and look for features that show up near the top across the board.

Wrapper Methods: Model-Guided Selection

Wrapper methods take the long road. They train a model again and again on different sets of features and watch which features actually move the needle. Recursive Feature Elimination (RFE) is the classic example: it starts with every feature, trains a model, kicks out the weakest one, then trains again, and keeps going until only the number you asked for is left. It is like cutting a team down to its starting lineup by playing match after match and dropping the weakest player each round.

📄 rfe.py: Recursive Feature Elimination

from sklearn.datasets import make_classification
from sklearn.feature_selection import RFE, RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = make_classification(n_samples=500, n_features=20, n_informative=10,
                            n_redundant=3, random_state=42)
feature_names = [f"feat_{i}" for i in range(20)]

# RFE: select exactly 10 features
model = RandomForestClassifier(n_estimators=100, random_state=42)
rfe = RFE(estimator=model, n_features_to_select=10, step=1)
rfe.fit(X, y)

selected = np.array(feature_names)[rfe.support_]
rankings = dict(zip(feature_names, rfe.ranking_))

print("RFE Selected (10 features):")
print(f"  {selected.tolist()}")
print(f"\nFeature Rankings (1=selected, higher=eliminated earlier):")
for feat in sorted(rankings, key=rankings.get):
    tag = "  <-- kept" if rankings[feat] == 1 else ""
    print(f"  {feat}: rank {rankings[feat]}{tag}")

# Compare accuracy with all features vs selected
all_score = cross_val_score(model, X, y, cv=5).mean()
sel_score = cross_val_score(model, X[:, rfe.support_], y, cv=5).mean()
print(f"\nAll 20 features: {all_score:.3f}")
print(f"RFE 10 features: {sel_score:.3f}")

▶ Output

RFE Selected (10 features):
  ['feat_0', 'feat_3', 'feat_4', 'feat_5', 'feat_7', 'feat_9', 'feat_12', 'feat_15', 'feat_18', 'feat_19']

Feature Rankings (1=selected, higher=eliminated earlier):
  feat_0: rank 1  <-- kept
  feat_3: rank 1  <-- kept
  feat_4: rank 1  <-- kept
  feat_5: rank 1  <-- kept
  feat_7: rank 1  <-- kept
  feat_9: rank 1  <-- kept
  feat_12: rank 1  <-- kept
  feat_15: rank 1  <-- kept
  feat_18: rank 1  <-- kept
  feat_19: rank 1  <-- kept
  feat_13: rank 2
  feat_14: rank 3
  feat_2: rank 4
  feat_17: rank 5
  feat_10: rank 6
  feat_16: rank 7
  feat_1: rank 8
  feat_8: rank 9
  feat_11: rank 10
  feat_6: rank 11

All 20 features: 0.876
RFE 10 features: 0.882

What happened here: RFE ranked every feature and kept its top 10. The model trained on those 10 features actually scored a touch higher than the model with all 20 (0.882 versus 0.876). That tiny bump is the whole point: cutting the noise let the model focus, so accuracy went up while the feature count went down. The ranking column tells the elimination story in reverse. Rank 1 means a feature survived to the end, and the bigger the rank, the earlier it got dropped (feat_6 at rank 11 was the very first to go).

Do not expect the kept set to be a neat feat_0 through feat_9 block. The useful columns are scattered, which is exactly why we let the algorithm pick instead of guessing by eye.

Embedded Methods: Built Into the Algorithm

Embedded methods are the practical sweet spot. The selection happens inside the model as it trains, so you do not pay for a separate step. An L1-penalized model pushes weak feature weights all the way to zero, and a tree-based model hands you an importance score for every feature once it has finished learning. It is like a chef tasting the soup while cooking and adjusting the seasoning in the same pot, instead of cooking first and then deciding what to fix. For day-to-day Python feature selection, this is the mode you will reach for most.

📄 embedded.py: L1 logistic regression and tree-based importance

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
import pandas as pd

X, y = make_classification(n_samples=500, n_features=20, n_informative=10,
                            n_redundant=3, random_state=42)
feature_names = [f"feat_{i}" for i in range(20)]

# 1. L1-penalized Logistic Regression: drives weak feature weights to exactly zero
# scikit-learn 1.9: use l1_ratio=1 for pure L1 (the old penalty="l1" is deprecated)
X_scaled = StandardScaler().fit_transform(X)
l1_model = LogisticRegression(solver="saga", l1_ratio=1, C=0.1,
                              max_iter=5000, random_state=42)
l1_model.fit(X_scaled, y)

l1_importance = pd.Series(np.abs(l1_model.coef_[0]), index=feature_names)
nonzero = l1_importance[l1_importance > 0].sort_values(ascending=False)
print(f"L1 Logistic Regression: {len(nonzero)} features with non-zero weights (out of 20)")
for feat, coef in nonzero.head(5).items():
    print(f"  {feat}: {coef:.4f}")

# 2. Random Forest feature importance
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)

rf_importance = pd.Series(rf.feature_importances_, index=feature_names)
rf_top = rf_importance.sort_values(ascending=False)
print(f"\nRandom Forest Feature Importance (top 10):")
for feat, imp in rf_top.head(10).items():
    print(f"  {feat}: {imp:.4f}")

▶ Output

L1 Logistic Regression: 10 features with non-zero weights (out of 20)
  feat_12: 0.9786
  feat_4: 0.6586
  feat_7: 0.5809
  feat_19: 0.5559
  feat_0: 0.3778

Random Forest Feature Importance (top 10):
  feat_12: 0.1555
  feat_0: 0.1281
  feat_4: 0.0729
  feat_5: 0.0641
  feat_7: 0.0635
  feat_19: 0.0583
  feat_15: 0.0581
  feat_18: 0.0565
  feat_3: 0.0437
  feat_9: 0.0421

What happened here: The L1 penalty drove 10 of the 20 weights to exactly zero, so the model effectively selected 10 features on its own while it was learning. Random Forest took a different route: it ranked every feature by how much it helped split the data into clean groups. The encouraging part is that both methods, despite working in completely different ways, put the same features near the top (feat_12, feat_4, feat_7, feat_19, and feat_0 all show up in both lists). When two unrelated methods agree, you can trust those features. That agreement is why embedded methods are the everyday workhorse: you get the selection baked in for free.

Common Mistakes

The single biggest mistake is letting your feature selector peek at the test data. Picture a student who gets to see the exam questions the night before: the marks look brilliant, but they prove nothing about what the student actually knows. Feature selection on the full dataset does the same trick. If the selector looks at every row before you split, it has already learned a little about the rows it is supposed to be tested on. Your scores then look great in your notebook and fall apart in production. This is called data leakage, and it is the quiet killer of machine learning projects.

❌ Mistake: Doing feature selection on the entire dataset

# BAD: Feature selection sees test data
# selector.fit(X_all, y_all)   leaks information from the test set
# X_train_selected = selector.transform(X_train)

# GOOD: Feature selection only on training data
# selector.fit(X_train, y_train)
# X_train_selected = selector.transform(X_train)
# X_test_selected = selector.transform(X_test)  # same features selected
print("Feature selection is part of the model. It must be done")
print("inside the cross-validation loop, not before splitting.")
print("Use Pipeline to automate this and prevent leakage.")

▶ Output

Feature selection is part of the model. It must be done
inside the cross-validation loop, not before splitting.
Use Pipeline to automate this and prevent leakage.

The fix is a Pipeline. Rahul, a 24-year-old analyst, has 20 columns of customer data and wants the 8 that best predict churn. He bundles scaling, selection, and the model into one Pipeline. Now scikit-learn runs the selection step separately inside each cross-validation fold, so the held-out fold stays truly unseen and his score is honest.

📄 pipeline.py: leak-free selection with a Pipeline

from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import cross_val_score

# Rahul has 20 columns of customer data and wants the 8 that predict churn.
X, y = make_classification(n_samples=500, n_features=20, n_informative=8,
                            n_redundant=2, random_state=42)

# The whole recipe lives in one Pipeline: scale -> pick 8 features -> classify.
# Selection runs inside each cross-validation fold, so the test fold stays unseen.
pipe = Pipeline([
    ("scale", StandardScaler()),
    ("select", SelectKBest(f_classif, k=8)),
    ("model", LogisticRegression(max_iter=1000, random_state=42)),
])

scores = cross_val_score(pipe, X, y, cv=5)
print(f"Pipeline accuracy (selection done per fold): {scores.mean():.3f}")
print("Per-fold scores: " + ", ".join(f"{s:.3f}" for s in scores))

▶ Output

Pipeline accuracy (selection done per fold): 0.718
Per-fold scores: 0.680, 0.700, 0.760, 0.730, 0.720

What happened here: Because the SelectKBest step lives inside the Pipeline, scikit-learn refits it on the training portion of every fold and never touches the test portion. The five fold scores hover around 0.72, which is a trustworthy estimate Rahul can actually report. Do the selection once on the full dataset by hand and those numbers would look better but lie to you. One Pipeline, zero leakage.

Practice Exercises

1. Exercise 1: Build a dataset with make_classification(n_features=30, n_informative=10) and run all three filter methods (variance, correlation, mutual information). List the features that show up in the top 10 of every method. Those are your safest keepers.
2. Exercise 2: Swap RFE for RFECV on the same data. RFECV uses cross-validation to pick the best number of features for you. Print how many features it chose and compare its accuracy against the fixed-10 RFE from this post.
3. Exercise 3: Take the leak-free Pipeline from the Common Mistakes section, then deliberately move the selection step outside the cross-validation loop (fit it once on all the data). Compare the two accuracy numbers and see how much the leaky version inflates the score.

Decision Summary

Here is the 30-second version of Python feature selection. Pin it.

• Use filter methods when you have a mountain of features (think 1000+) and need a fast first cut. They cost almost nothing to run, so start here to throw out the obvious junk.
• Use wrapper methods (RFE) when your dataset is small to medium and you want the most accurate feature set, and you can spare the compute. They are slow because they train many models, so save them for when accuracy matters more than speed.
• Use embedded methods (L1 or tree importance) for almost everything else. They give you selection and a trained model in one shot, which makes them the practical default for day-to-day work.
• Always wrap selection in a Pipeline so it runs inside cross-validation. This is not optional. It is the one rule that keeps your scores honest.

Conclusion

You now have the full toolkit for Python feature selection. Filter methods score each column on its own and give you a fast first pass. Wrapper methods like RFE train a model over and over to find the sharpest subset. Embedded methods, L1 logistic regression and tree importance, bake the selection right into training, which is why they are the everyday default. Above all, you learned the one non-negotiable rule: wrap selection inside a Pipeline so it runs per fold and your scores stay honest.

Next up is Linear Regression, Theory and Implementation, where you put a trimmed feature set to work in your first real model. For the full roadmap from beginner basics to production machine learning, head back to the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Feature Selection

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: What is the difference between filter, wrapper, and embedded feature selection methods?

Filter methods score each feature independently using statistics such as correlation, variance, or mutual information, so they are fast but blind to how features interact. Wrapper methods like RFE train a model repeatedly on different feature subsets and keep the set that scores best, which is accurate but slow. Embedded methods, such as L1 (Lasso) logistic regression or tree importance, perform selection while the model trains, giving a good balance of speed and quality.

Q: A colleague selects the top 20 features on the full dataset, then cross-validates and reports 95% accuracy. What went wrong, and where should selection have happened?

If you select features on the whole dataset before splitting, the selector has already seen the test rows, which leaks information and inflates your scores. Wrapping the selector in a scikit-learn Pipeline forces it to refit on only the training portion of each fold, so the held-out fold stays truly unseen. The reported accuracy then reflects real-world performance instead of a number that collapses in production.

Q: How does L1 (Lasso) regularization perform feature selection?

An L1 penalty adds the sum of the absolute values of the coefficients to the loss function. This penalty pushes weak or useless feature weights all the way to exactly zero, so those features drop out of the model automatically. In scikit-learn 1.9 you get pure L1 with LogisticRegression(solver="saga", l1_ratio=1), and the features left with non-zero coefficients are your selected set.

Q: Does removing features always improve accuracy?

Not always. Removing irrelevant and redundant features usually helps linear models and distance-based models like K-Nearest Neighbors (KNN), and it always speeds up training and simplifies the model. Tree-based models such as Random Forest and XGBoost are more tolerant of noise because they naturally ignore unhelpful features during splits. Cut too aggressively and you can drop informative features and lose accuracy, so validate the trade-off with cross-validation.

Q: Scenario: you have 2,000 features and only 3 hours of compute. What selection strategy do you pick?

Start with cheap filter methods to cut the obvious junk fast: drop near-constant columns with a variance threshold, remove highly correlated duplicates, then rank the rest with mutual information or the ANOVA F-test to get down to a few hundred candidates. Wrapper methods like RFE would be far too slow to train thousands of models at this scale. Finish with an embedded method (L1 or tree importance) on the reduced set to get your final features plus a trained model in a single pass.

Q: Scenario: your model scores 0.95 in the notebook but only 0.70 in production. Feature selection is in the pipeline. What do you check first?

The gap is a classic sign of data leakage during selection. Check whether the selector was fit on the entire dataset before the train/test split instead of inside each cross-validation fold. Confirm the selection step lives inside a Pipeline passed to cross_val_score so it refits per fold. Also verify the target column, or anything derived from it, did not sneak into the feature matrix, since a leaked target produces exactly this kind of too-good-to-be-true score.

Q: When two different methods rank features differently, which one do you trust?

Do not trust a single method blindly, because each measures a different kind of usefulness. Filter scores miss feature interactions, and one model's importance is specific to that model. The practical move is to run two or three methods and trust the features that rise to the top across all of them, since agreement between unrelated methods is strong evidence a feature genuinely matters.

Q: How is feature selection different from dimensionality reduction like Principal Component Analysis (PCA)?

Feature selection keeps a subset of your original columns, so the result stays interpretable: you know exactly which real features survived. PCA and similar techniques build brand new features from linear combinations of the originals, which compress the data well but are hard to explain because each new component mixes many columns together. Choose selection when interpretability matters and reduction when you only care about compact, predictive signal.

Series: Python + AI/ML Cookbook, Part 5: Machine Learning

Further reading: for the full reference, see scikit-learn documentation.

Related Posts

Previous: ML: Feature Engineering, Turning Raw Data into Predictive Power

Next: ML: The Math Behind Machine Learning, Intuition Before Formulas

Series Home: Python + AI/ML Tutorial Series