You build a model, it nails every point in your training data, then it falls apart the moment fresh numbers show up. That gap between the practice test and the real exam is exactly what ridge lasso regression is built to close. This guide starts with polynomial regression for curved data, then leans on Ridge (L2) and Lasso (L1) to rein in that overfitting, with every example run on scikit-learn.

“The art of statistical learning is finding the right level of model complexity.”

Trevor Hastie, Elements of Statistical Learning

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Advanced | Reading Time: 14 minutes

When the link between your features and your target is not a straight line, plain linear regression hits a wall. Think about house prices in Pune. A 1,000 sqft flat might cost around 50 lakh, but a 2,000 sqft flat does not simply cost double. Bigger homes carry a premium, so the price curves upward instead of rising in a straight line. Polynomial regression bends to fit that curve by adding squared, cubed, and interaction terms to the model.

But polynomial models have a dark side. They overfit easily. A high-degree polynomial will happily twist itself through every point, including the noise, like a student who memorizes the answer key instead of learning the subject. Great on the practice test, useless on the real exam. Regularization is the cure. Ridge regression (L2) keeps the coefficients small. Lasso regression (L1) goes one step further and pushes some coefficients all the way to zero, which quietly does feature selection for you. Elastic Net blends both.

Understanding when to use each technique is more important than memorizing the formulas, and that judgment is what a solid ridge lasso regression workflow rests on. This post builds it.

Prerequisites

Polynomial Regression: Fitting Curves

The idea is simple. Take your single feature, also raise it to the power of 2, 3, and so on, then fit a normal linear model on top of those new columns. The model is still linear in its coefficients, so all the old machinery works, but now it can draw a curve instead of a line. The catch is choosing how high to go. Too low and the curve stays too stiff to follow the data (underfitting). Too high and it wiggles to chase every random bump (overfitting). The script below walks the degree from 1 up to 20 and lets cross-validation tell us where the sweet spot sits.

📄 polynomial_regression.py: from underfitting to overfitting

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(42)

# True relationship: y = 2x² - 3x + 10 + noise
X = rng.uniform(-3, 3, 100).reshape(-1, 1)
y = 2 * X.ravel()**2 - 3 * X.ravel() + 10 + rng.normal(0, 2, 100)

print(f"{'Degree':>6} | {'CV R²':>10} | {'Assessment'}")
print("-" * 45)

for degree in [1, 2, 3, 5, 10, 20]:
    model = Pipeline([
        ("poly", PolynomialFeatures(degree=degree)),
        ("reg", LinearRegression())
    ])
    scores = cross_val_score(model, X, y, cv=5, scoring="r2")
    mean_r2 = scores.mean()

    if mean_r2 < 0.5:
        assessment = "Underfitting"
    elif scores.std() > 0.1:
        assessment = "Overfitting"
    else:
        assessment = "Good fit" if mean_r2 > 0.9 else "Moderate"

    print(f"{degree:>6} | {mean_r2:>9.4f} | {assessment}")

▶ Output

Degree |      CV R² | Assessment
---------------------------------------------
     1 |    0.4876 | Underfitting
     2 |    0.9299 | Good fit
     3 |    0.9319 | Good fit
     5 |    0.9308 | Good fit
    10 |    0.9210 | Good fit
    20 |   -0.1399 | Underfitting

What happened here: Degree 1 is a straight line, so it underfits our curved data and scores a weak 0.49. Degree 2 jumps to 0.93, which makes sense because the true relationship really is quadratic. Degrees 3, 5, and even 10 stay close behind because the extra terms mostly stay quiet and do not hurt much. Then degree 20 falls off a cliff to a negative score. A negative R-squared means the model does worse than just guessing the average every time. (Our little label rule prints “Underfitting” there only because it checks the low-score case first; the real story at degree 20 is wild overfitting that cross-validation then punishes.) This whole table is the bias-variance tradeoff in one screen.

Ridge Regression (L2): Shrinking Coefficients

Ridge adds one extra term to the cost the model is trying to minimize: the usual error (mean squared error, or MSE) plus alpha times the sum of the squared weights. That second term acts like a tax on big coefficients, so the model would rather spread its bets across many small weights than lean hard on a few huge ones. Picture a household budget where every rupee of spending is taxed. You do not stop spending, you just keep each line item modest. The alpha knob sets the tax rate. At alpha=0 there is no tax and you are back to ordinary linear regression. Crank alpha up toward infinity and every weight gets squeezed toward zero.

📄 ridge.py: Ridge regression taming polynomial overfitting

import numpy as np
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import Ridge, RidgeCV
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(42)
X = rng.uniform(-3, 3, 100).reshape(-1, 1)
y = 2 * X.ravel()**2 - 3 * X.ravel() + 10 + rng.normal(0, 2, 100)

# Degree 10 polynomial WITHOUT regularization
unreg = Pipeline([
    ("poly", PolynomialFeatures(degree=10)),
    ("scaler", StandardScaler()),
    ("reg", Ridge(alpha=0))  # alpha=0 = no regularization
])
scores_unreg = cross_val_score(unreg, X, y, cv=5, scoring="r2")

# Degree 10 polynomial WITH Ridge regularization
for alpha in [0.01, 0.1, 1.0, 10.0, 100.0]:
    ridge_pipe = Pipeline([
        ("poly", PolynomialFeatures(degree=10)),
        ("scaler", StandardScaler()),
        ("reg", Ridge(alpha=alpha))
    ])
    scores = cross_val_score(ridge_pipe, X, y, cv=5, scoring="r2")
    print(f"Ridge alpha={alpha:>6}: R²={scores.mean():.4f} ± {scores.std():.4f}")

# Automatic alpha selection with RidgeCV
best_pipe = Pipeline([
    ("poly", PolynomialFeatures(degree=10)),
    ("scaler", StandardScaler()),
    ("reg", RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]))
])
best_pipe.fit(X, y)
print(f"\nRidgeCV chose alpha={best_pipe.named_steps['reg'].alpha_}")
print(f"No regularization (alpha=0): R²={scores_unreg.mean():.4f}")

▶ Output

Ridge alpha=  0.01: R²=0.9255 ± 0.0127
Ridge alpha=   0.1: R²=0.9265 ± 0.0129
Ridge alpha=   1.0: R²=0.9281 ± 0.0136
Ridge alpha=  10.0: R²=0.9195 ± 0.0230
Ridge alpha= 100.0: R²=0.7992 ± 0.0405

RidgeCV chose alpha=1.0
No regularization (alpha=0): R²=0.9210

What happened here: With no penalty (alpha=0) the degree-10 polynomial scored 0.9210. A light touch of Ridge at alpha=1.0 nudged it up to 0.9281 by reining in the wild coefficients. Notice the two extremes. Too little penalty and you keep some overfitting. Too much (alpha=100) and the score crashes to 0.7992 because Ridge has flattened the useful coefficients along with the harmful ones. RidgeCV tried each alpha for us and landed on 1.0. The key idea to carry forward: Ridge never deletes a feature. It just turns the volume down on all of them at once, so every feature still has a say, only a quieter one.

Lasso Regression (L1): Feature Selection Built In

Lasso swaps the squared penalty for an absolute-value one: MSE plus alpha times the sum of the absolute weights. That small change has a big effect. Instead of just shrinking weights toward zero, Lasso can set some of them to exactly zero, which means it throws those features out of the model entirely. The geometry behind it is the diamond-shaped L1 constraint, whose sharp corners sit right on the axes, so the best solution tends to land on a corner where one or more weights are zero.

The practical payoff is that Lasso does feature selection for free. Think of packing one carry-on bag for a trip: the size limit forces you to leave the items you do not really need behind, so only the essentials make the cut.

The diagram compares Ridge (L2) and Lasso (L1) regularization. Both add a penalty to the loss that discourages large coefficients, but Ridge only shrinks coefficients toward zero while Lasso can push them exactly to zero. That single difference is what turns Lasso into a feature selection tool, because any coefficient that hits zero is effectively dropped from the model. Elastic Net sits in the middle and combines both penalties, giving you a tunable balance between the stability of Ridge and the sparsity of Lasso. The code examples below show how the alpha parameter controls this trade-off.

📄 lasso.py: Lasso zeroing out features

import numpy as np
from sklearn.linear_model import Lasso, LassoCV
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression

# 20 features, only 5 are truly informative
X, y = make_regression(n_samples=200, n_features=20, n_informative=5,
                        noise=10, random_state=42)
X_scaled = StandardScaler().fit_transform(X)

print(f"{'Alpha':>8} | {'Non-zero':>8} | {'R²':>8}")
print("-" * 32)

for alpha in [0.01, 0.1, 1.0, 2.0, 10.0]:
    lasso = Lasso(alpha=alpha, max_iter=10000)
    lasso.fit(X_scaled, y)
    nonzero = np.sum(lasso.coef_ != 0)
    r2 = lasso.score(X_scaled, y)
    print(f"{alpha:>8.3f} | {nonzero:>8} | {r2:>7.4f}")

# Auto-select best alpha
lasso_cv = LassoCV(cv=5, random_state=42, max_iter=10000)
lasso_cv.fit(X_scaled, y)
print(f"\nLassoCV: alpha={lasso_cv.alpha_:.4f}, non-zero={np.sum(lasso_cv.coef_ != 0)}")
print(f"Selected features: {np.where(lasso_cv.coef_ != 0)[0]}")

▶ Output

   Alpha | Non-zero |       R²
--------------------------------
   0.010 |       20 |  0.9961
   0.100 |       19 |  0.9960
   1.000 |        7 |  0.9955
   2.000 |        5 |  0.9948
  10.000 |        4 |  0.9720

LassoCV: alpha=0.4513, non-zero=11
Selected features: [ 0  1  4  5  7  8  9 10 11 13 18]

What happened here: As alpha grows, Lasso trims more and more features. At alpha=2.0 exactly 5 features survive, and they turn out to be the precise 5 the data generator marked as informative. That is feature selection working perfectly. Push to alpha=10.0 and Lasso gets greedy, dropping to 4 features and starting to lose signal, so the R-squared dips to 0.9720. There is a subtle lesson in the last two lines.

LassoCV did not pick the sparse alpha=2.0. It chose alpha=0.4513 and kept 11 features, because cross-validation here is chasing the best prediction score, not the smallest model. If your real goal is a short, readable list of features, do not blindly trust the CV alpha. Look at the path and pick the point where extra features stop earning their keep.

Elastic Net: Best of Both Worlds

Elastic Net mixes the two penalties in one model. You get the corner-cutting sparsity of Lasso and the gentle, group-friendly shrinking of Ridge, dialed by a single number called l1_ratio. At l1_ratio=0 it is pure Ridge, at l1_ratio=1 it is pure Lasso, and anything in between is a blend. This matters most when features come in correlated groups. Lasso, faced with two near-twin features, tends to keep one at random and drop the other. Elastic Net is happy to keep both, the way a coach keeps two equally good players instead of cutting one on a coin flip. That flexibility is why it sits beside Ridge and Lasso regression in scikit-learn’s standard lineup.

📄 elastic_net.py: combining L1 and L2

from sklearn.linear_model import ElasticNet, ElasticNetCV
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_regression
import numpy as np

X, y = make_regression(n_samples=200, n_features=20, n_informative=5,
                        noise=10, random_state=42)
X_scaled = StandardScaler().fit_transform(X)

# l1_ratio: 0 = pure Ridge, 1 = pure Lasso, 0.5 = equal mix
for l1_ratio in [0.1, 0.5, 0.9]:
    en = ElasticNet(alpha=1.0, l1_ratio=l1_ratio, max_iter=10000)
    en.fit(X_scaled, y)
    nonzero = np.sum(en.coef_ != 0)
    print(f"l1_ratio={l1_ratio}: {nonzero} non-zero features, R²={en.score(X_scaled, y):.4f}")

# Auto-select both alpha and l1_ratio
en_cv = ElasticNetCV(cv=5, l1_ratio=[0.1, 0.3, 0.5, 0.7, 0.9], random_state=42, max_iter=10000)
en_cv.fit(X_scaled, y)
print(f"\nElasticNetCV: alpha={en_cv.alpha_:.4f}, l1_ratio={en_cv.l1_ratio_}")
print(f"Non-zero: {np.sum(en_cv.coef_ != 0)}, R²={en_cv.score(X_scaled, y):.4f}")

▶ Output

l1_ratio=0.1: 20 non-zero features, R²=0.7887
l1_ratio=0.5: 19 non-zero features, R²=0.8910
l1_ratio=0.9: 12 non-zero features, R²=0.9863

ElasticNetCV: alpha=0.1242, l1_ratio=0.9
Non-zero: 20, R²=0.9959

What happened here: l1_ratio slides between Ridge (0, all L2) and Lasso (1, all L1). At l1_ratio=0.1 the model leans Ridge, so it keeps all 20 features because L2 rarely zeros anything out. At l1_ratio=0.9 it leans Lasso and trims down to 12. Watch the scores too. With a fixed, fairly strong alpha=1.0, the Ridge-heavy end actually scores lower (0.7887) because that much L2 over-shrinks the real signal, while the Lasso-heavy end keeps the score high.

Then ElasticNetCV searched alpha and l1_ratio together and chose a much gentler alpha=0.1242, which is why it ends up keeping all 20 features with the best R-squared of all (0.9959). The takeaway: reach for Elastic Net when your features come in correlated clusters, the exact case where plain Lasso gets twitchy and drops members of a group at random.

Common Ridge Lasso Regression Mistakes

❌ Mistake: Forgetting to scale features before regularization

# Ridge and Lasso penalize large coefficients
# If feature A ranges 0-1 and feature B ranges 0-1000000,
# the penalty hits feature B's coefficient much harder
# This means unscaled features get unfair regularization

# WRONG: features not scaled
from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)
model.fit(X_train, y_train)  # Penalty is unfair to large-scale features

# RIGHT: always scale before regularization
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("ridge", Ridge(alpha=1.0))
])
pipeline.fit(X_train, y_train)  # Fair penalty on all features

Practice Exercises

1. Exercise 1: Take the polynomial script and plot CV R-squared against degree for degrees 1 to 15. Mark where overfitting starts. Does the curve match the table in this post?
2. Exercise 2: On the same make_regression data, build a small loop that prints, for each alpha, how many of the 5 true informative features Lasso actually keeps. Find the alpha range where it recovers exactly those 5.
3. Exercise 3: Generate a dataset with two highly correlated features (copy one column and add tiny noise), then compare what Lasso versus Elastic Net does to that pair. Confirm that Lasso drops one while Elastic Net keeps both.

Conclusion

You now have the full ridge lasso regression toolkit for taming model complexity. Polynomial features let a linear model bend into curves, Ridge (L2) shrinks every coefficient to fight overfitting, Lasso (L1) shrinks and drops features so you get selection for free, and Elastic Net blends the two for correlated groups. The one habit that ties it all together: always scale your features first, then let cross-validation choose alpha (and l1_ratio) instead of guessing. Next we move from predicting numbers to predicting categories with logistic regression. For the full learning path, visit the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Ridge and Lasso Regression

Interviewers rarely ask for definitions. They ask what happens in situations like these.

Q: Why must you standardize features before Ridge or Lasso, when scaling is optional for plain linear regression?

Regularization penalizes the size of the coefficients, and a coefficient’s size depends on its feature’s units. Left unscaled, a feature measured in millions earns a tiny coefficient and is barely penalized, while a 0 to 1 feature gets hit hard, so the penalty is unfair. Plain OLS has no penalty term, so rescaling a feature only rescales its coefficient without changing predictions. Put a StandardScaler before the model inside a Pipeline to keep the fit and any cross-validation honest.

Q: What is the geometric reason Lasso produces exact zeros while Ridge does not?

The L1 constraint region is a diamond with sharp corners sitting on the axes, and the loss contours usually first touch that region at a corner, where one or more coefficients are exactly zero. Ridge’s L2 constraint is a smooth circle or sphere with no corners, so the optimum lands off the axes and coefficients shrink toward zero but never reach it. That single shape difference is why Lasso doubles as a feature selector and Ridge does not.

Q: How do alpha and l1_ratio interact in Elastic Net?

Alpha sets the overall strength of the penalty, while l1_ratio splits that penalty between L1 (sparsity) and L2 (shrinkage). l1_ratio=0 is pure Ridge and l1_ratio=1 is pure Lasso, with blends in between. You tune both together with ElasticNetCV, because a strong alpha with a Lasso-heavy ratio can over-prune the real signal, while a gentle alpha keeps more features and often scores higher.

Q: Your Lasso model’s selected features change completely between two runs on nearly identical data. What is happening and how do you stabilize it?

Lasso is unstable when features are correlated: it tends to keep one member of a correlated group and zero the rest, so a small change in the data flips which one survives. Switch to Elastic Net with an l1_ratio around 0.5 so correlated features are kept or dropped together, and use ElasticNetCV with a fixed random_state and enough CV folds. If you need a defensible feature list, confirm it with a resampling or stability-selection style check rather than a single fit.

Q: A teammate reports that LassoCV kept 11 features but only 5 were expected to matter. Is the model broken?

No. LassoCV optimizes the cross-validated prediction score, not the smallest feature set, so it often keeps extra weakly useful features. If sparsity is the real goal, inspect the coefficient path and pick a larger alpha where the score barely drops, or apply the one standard error rule to choose the simplest model within one standard error of the best score. The CV alpha is a starting point, not the final word.

Q: Your polynomial regression scores great on training data but a negative R-squared on the test fold. What do you check first?

A negative test R-squared means the model predicts worse than simply guessing the mean, the classic signature of severe overfitting from too high a polynomial degree. Check the degree first and lower it, then add Ridge or Lasso on top of the polynomial features, and make sure a StandardScaler runs before the regularizer. Always judge the degree by the cross-validated score, never by the training score.

Series: Python + AI/ML Cookbook (Part 5: Machine Learning)

Further reading: scikit-learn documentation is the authoritative source on this.

Related Posts

Previous: ML: Linear Regression, Theory and Implementation

Next: ML: Logistic Regression, Binary and Multi-class

Series Home: Python + AI/ML Tutorial Series