Default settings got your model to 90%, and the last few points are hiding in knobs the model never learns on its own. Hyperparameter tuning is how you find them without guessing. This post compares three search strategies, GridSearchCV, RandomizedSearchCV, and Bayesian optimization with Optuna, all with real tested numbers, so you can pick the right one without burning compute.

“Spend your compute budget exploring more of the search space, not every cell of a grid nobody validated.”

The takeaway from Bergstra & Bengio, “Random Search for Hyper-Parameter Optimization” (2012)

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0, Optuna 4.9.0 | Difficulty: Advanced | Reading Time: 10 minutes

A model running on default hyperparameters is like a car stuck in second gear. It still drives, but you are leaving real performance on the table. Things like the number of trees in a random forest, the learning rate of gradient boosting, or the regularization strength of logistic regression are not learned from your data. You set them by hand before training starts, and they decide whether your model underfits, overfits, or lands right in the sweet spot.

Think of it like tuning a guitar. The blunt way is to try every peg position one by one until something sounds right. That is grid search: try every combination. A grid of 4 parameters with 5 values each is 625 settings, and with 5-fold cross-validation that becomes 3,125 separate trainings. Random search instead picks settings at random and usually lands on results just as good in a fraction of the time. Bayesian optimization goes one step smarter: it listens to how each attempt sounded and tunes toward the promising notes, spending its next guesses where the score is already trending up.

Prerequisites

Three Tuning Strategies

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram lays out the three hyperparameter tuning strategies side by side. Grid search checks every combination in the grid, so it is thorough but slow. Random search samples combinations from your defined ranges, and it is surprisingly effective because it tries more unique values of each hyperparameter instead of repeating the same few. Bayesian optimization uses past results to pick the next configuration on purpose, which makes it converge fastest when each trial is expensive. The examples below run all three on the very same model so you can compare the real numbers.

📄 grid_vs_random.py: GridSearchCV vs RandomizedSearchCV

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import time
import numpy as np

X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [5, 10, 20, None],
    "min_samples_split": [2, 5, 10],
    "min_samples_leaf": [1, 2, 4],
}

# Grid Search: try EVERY combination
start = time.time()
grid = GridSearchCV(RandomForestClassifier(random_state=42), param_grid,
                    cv=3, scoring="f1", n_jobs=-1)
grid.fit(X, y)
grid_time = time.time() - start

# Random Search: sample 30 random combinations
start = time.time()
rand = RandomizedSearchCV(RandomForestClassifier(random_state=42), param_grid,
                          n_iter=30, cv=3, scoring="f1", random_state=42, n_jobs=-1)
rand.fit(X, y)
rand_time = time.time() - start

total_combos = 3 * 4 * 3 * 3
print(f"Total combinations: {total_combos}")
print(f"Grid Search:   {grid_time:.1f}s, best F1={grid.best_score_:.4f}")
print(f"Random Search: {rand_time:.1f}s, best F1={rand.best_score_:.4f} (30 of {total_combos} tried)")
print(f"Speedup: {grid_time/rand_time:.1f}x")
print(f"\nGrid best params:   {grid.best_params_}")
print(f"Random best params: {rand.best_params_}")

▶ Output

Total combinations: 108
Grid Search:   46.8s, best F1=0.8918
Random Search: 12.1s, best F1=0.8895 (30 of 108 tried)
Speedup: 3.9x

Grid best params:   {'max_depth': 20, 'min_samples_leaf': 1, 'min_samples_split': 5, 'n_estimators': 50}
Random best params: {'n_estimators': 200, 'min_samples_split': 2, 'min_samples_leaf': 1, 'max_depth': None}

What happened here: Random search landed almost on top of grid search (F1=0.8895 versus 0.8918, a gap of about two thousandths) while trying only 30 of the 108 combinations, so it finished in roughly a quarter of the time. The exact seconds and the speedup will shift from run to run and machine to machine, but the pattern holds: you pay a tiny bit of score for a big chunk of saved time. As the grid grows into hundreds of thousands of combos, that saving turns dramatic. The Bergstra and Bengio 2012 paper showed why random search is more efficient: not all hyperparameters matter equally, and sampling randomly explores more distinct values of the ones that do.

Bayesian Optimization with Optuna

Picture hunting for the best chai stall in a new city. Grid search would walk down every street and stop at every stall. Random search would wander into a handful of stalls at random. Bayesian optimization asks a few people, notices the good ones cluster near the old market, and points its next steps there. That is exactly what Optuna does: each trial updates a model of where good scores are likely to hide, and the next configuration is chosen to chase that hunch instead of guessing blind.

📄 optuna_tuning.py: smart search that learns from results

# pip install optuna
import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

def objective(trial):
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 50, 300),
        "max_depth": trial.suggest_int("max_depth", 3, 30),
        "min_samples_split": trial.suggest_int("min_samples_split", 2, 20),
        "min_samples_leaf": trial.suggest_int("min_samples_leaf", 1, 10),
    }
    model = RandomForestClassifier(**params, random_state=42)
    score = cross_val_score(model, X, y, cv=3, scoring="f1").mean()
    return score

optuna.logging.set_verbosity(optuna.logging.WARNING)
# Seed the sampler so the run is reproducible
sampler = optuna.samplers.TPESampler(seed=42)
study = optuna.create_study(direction="maximize", sampler=sampler)
study.optimize(objective, n_trials=50)

print(f"Best F1: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")
print(f"Trials: {len(study.trials)}")

▶ Output

Best F1: 0.8905
Best params: {'n_estimators': 224, 'max_depth': 25, 'min_samples_split': 5, 'min_samples_leaf': 1}
Trials: 50

What happened here: Optuna scored F1=0.8905 in 50 trials, landing right between random search (0.8895) and grid search (0.8918) on this small toy dataset. It did not beat grid search here, and that is the honest result: when the problem is easy and the grid is already decent, all three strategies cluster within a whisker of each other. The real win is in the search itself.

Optuna explored a continuous range and learned from each trial which regions looked promising, so it tried values the fixed grid never offered, like n_estimators=224 instead of just 50, 100, or 200. That smart search pays off most when each trial is expensive (deep learning, large datasets, big models), where you cannot afford to fit hundreds of combinations.

Note the seed: without TPESampler(seed=42) the sampler picks different points each run, so your best score and params would shift every time.

Common Mistakes

The most expensive mistake in hyperparameter tuning is touching the test set too early. Think of the test set like a sealed exam paper. The moment you peek at it to choose your settings, the exam is no longer a fair measure of what you actually know. If you tune on the test set, your final score looks great on paper and then collapses on real, unseen data.

❌ Mistake: Tuning on the test set

# If you tune hyperparameters using the test set, your test score
# is no longer an honest estimate of real-world performance.
# The test set has leaked into your model selection process.
# Always tune with cross-validation on the TRAINING set.
# The test set is touched ONCE, at the very end.
print("Correct workflow:")
print("  1. Split: train + test")
print("  2. Tune: GridSearchCV on train (uses internal CV)")
print("  3. Evaluate: best model on test set (ONE time)")

▶ Output

Correct workflow:
  1. Split: train + test
  2. Tune: GridSearchCV on train (uses internal CV)
  3. Evaluate: best model on test set (ONE time)

What happened here: GridSearchCV and RandomizedSearchCV already run their own cross-validation internally, so they split the training data into folds and score on data the model has not seen during that fold. That keeps your real test set sealed. You fit the search on the training set, read off best_params_, train one final model with those settings, and only then unseal the test set for a single, honest score. Touch the test set once, at the very end.

Practice Exercises

1. Exercise 1: GridSearchCV for 2 Random Forest hyperparameters.
2. Exercise 2: Compare Grid vs Random search speed and results.
3. Exercise 3: Bayesian optimization with optuna. Compare convergence.

Conclusion

You now have three tools for squeezing real performance out of a model, and a feel for when each one earns its keep. Grid search is the exhaustive sweep: use it only for a tiny grid you can afford to fit in full. Random search samples across your ranges and lands on near-best results in a fraction of the time, which makes it the sensible default for most jobs. Bayesian optimization with Optuna learns from every trial and points its next guess at the promising regions, so it shines when each fit is expensive.

Just as important, you saw the one mistake that quietly ruins results: tuning on the test set. Keep that exam paper sealed until the very end, and let cross-validation do the choosing.

Next up, you will fold tuning into a full scikit-learn pipeline so preprocessing and model selection travel together without leaking data. For the complete learning path from Python basics through applied machine learning, visit the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Hyperparameter Tuning

These come from real screens and onsites. Practice answering before you read each answer.

Q: What is the difference between a parameter and a hyperparameter?

Parameters are learned from the data during training, like the split thresholds inside a decision tree or the weights of a linear model. Hyperparameters are set by you before training starts and control how the model learns, like the number of trees, the max depth, or the learning rate. You tune hyperparameters; the model fits parameters on its own.

Q: Why does random search often match grid search in far less time?

Because not all hyperparameters matter equally. Grid search wastes trials re-testing the same few values of the important parameter while the unimportant ones change. Random search samples many distinct values of every parameter, so it is more likely to hit a good value of the one that actually moves the score. Bergstra and Bengio (2012) showed this holds across many real problems.

Q: Why must you avoid tuning hyperparameters on the test set?

If you pick settings by looking at test scores, the test set has leaked into model selection and no longer measures unseen performance. Your reported score becomes optimistic and collapses on real data. Tune with cross-validation on the training set, then touch the test set exactly once for the final estimate.

Q: How does Bayesian optimization decide the next configuration to try?

It builds a probabilistic model of the score as a function of the hyperparameters, using the trials already run. It then picks the next point that balances exploring uncertain regions against exploiting regions that look promising. Optuna’s default TPE sampler does this, which is why it converges faster than blind sampling when each trial is expensive.

Q: Your GridSearchCV over a large grid has been running for hours and has not finished. What do you change first?

Switch to RandomizedSearchCV with a fixed n_iter budget, or to Optuna, so you cap the number of fits instead of trying every combination. Confirm n_jobs=-1 so all cores are used, and consider HalvingGridSearchCV, which throws out weak configurations early. Also shrink the grid: drop values that never win and narrow ranges around the region that looks best.

Q: A teammate reports a stellar cross-validation F1 during tuning, but the model performs far worse in production. What do you check first?

First check for data leakage: was scaling, encoding, or feature selection fit on the whole dataset before the split instead of inside each CV fold? Next confirm the test set was never used to choose hyperparameters. Then look at whether the CV folds respect the data structure, for example using time-aware or grouped splits when rows are not independent. Leakage during tuning is the most common cause of a great CV score that does not hold up.

Series: Python + AI/ML Cookbook. Part 5: Machine Learning

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: ML: Principal Component Analysis (PCA), Dimensionality Reduction Explained

Next: ML: Pipeline, End-to-End ML Pipelines with scikit-learn

Series Home: Python + AI/ML Tutorial Series