Train a single decision tree, swap out just a handful of training rows, and the whole thing can rearrange itself while your accuracy swings around. That jumpiness is the exact problem a random forest fixes: it grows hundreds of trees on different random slices of your data and lets them vote. In this guide you will see why a crowd of “good enough” trees beats one clever tree, and work through bagging, out-of-bag scoring, feature importance, and the handful of hyperparameters that really matter.

“Ask one friend where to eat and you might get a bad night. Ask a hundred and the average answer is usually pretty good.”

The wisdom of crowds, in one sentence

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Intermediate | Reading Time: 17 minutes

A single decision tree is fast and easy to read, but it is jumpy. Swap out a handful of training rows and the whole tree can rearrange itself. Random Forest fixes that by growing hundreds of trees, each one trained on a slightly different random slice of the data, and then letting them vote. Any one tree can be wrong. The majority vote, taken across the whole crowd, is usually right.

Think of a panel of doctors reading the same scan. One doctor might miss something on a bad day. But if ninety out of a hundred independent doctors say “this looks fine”, you trust that answer far more than any single opinion. Random Forest builds that panel for you, and the trick is making sure the doctors do not all think alike.

The “random” in Random Forest comes from two places. First, each tree learns from a bootstrap sample, that is, a random draw of rows taken with replacement, so every tree sees a different version of the data. Second, at each split point a tree is only allowed to look at a random handful of features, not all of them. Those two dice rolls force the trees to disagree, and they end up making different mistakes. Average a pile of different mistakes together and they mostly cancel out, which leaves the real signal behind.

Random Forest is the “just start here” algorithm. If you have a table of data and no idea which model to reach for first, reach for this one. It handles curvy non-linear patterns, mixes numbers and categories happily, needs no feature scaling, shrugs off outliers, and is genuinely hard to overfit. The one real downside is that you cannot eyeball it. Nobody can sketch 500 trees on a whiteboard and follow the logic by hand.

Prerequisites

How Random Forest Works

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The diagram shows how a random forest model builds its ensemble. Each decision tree trains on its own random slice of rows (bagging) and is only allowed to consider a random handful of features at each split. Then the trees vote: majority wins for classification, or you take the average for regression. That double dose of randomness is exactly what makes the forest steadier and more accurate than any one tree. Notice that no single tree has to be brilliant. The power comes from stacking a lot of “good enough” trees that each get things wrong in different spots, so their errors cancel instead of pile up.

📄 random_forest_basics.py: one tree versus a whole forest

import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

# random_state keeps the toy data and the trees reproducible
X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                            random_state=42)

# Single tree vs forest
single_tree = DecisionTreeClassifier(random_state=42)
forest = RandomForestClassifier(n_estimators=100, random_state=42)

tree_scores = cross_val_score(single_tree, X, y, cv=5)
forest_scores = cross_val_score(forest, X, y, cv=5)

print(f"Single Tree:   {tree_scores.mean():.3f} +/- {tree_scores.std():.3f}")
print(f"Random Forest: {forest_scores.mean():.3f} +/- {forest_scores.std():.3f}")
print(f"\nImprovement:   {(forest_scores.mean() - tree_scores.mean()):.3f}")

# How many trees do you need?
print(f"\n{'Trees':>6} | {'CV Score':>10} | {'Std':>8}")
print("-" * 30)
for n_trees in [1, 5, 10, 50, 100, 500]:
    rf = RandomForestClassifier(n_estimators=n_trees, random_state=42)
    scores = cross_val_score(rf, X, y, cv=5)
    print(f"{n_trees:>6} | {scores.mean():>9.3f} | {scores.std():>7.3f}")

▶ Output

Single Tree:   0.862 +/- 0.021
Random Forest: 0.910 +/- 0.030

Improvement:   0.048

 Trees |   CV Score |      Std
------------------------------
     1 |     0.814 |   0.038
     5 |     0.886 |   0.023
    10 |     0.906 |   0.029
    50 |     0.912 |   0.041
   100 |     0.910 |   0.030
   500 |     0.914 |   0.029

What happened here: The lone tree scored about 86%. The forest of 100 trees jumped to 91%, a free 4.8 point gain with no new data and no clever tuning, just by letting a crowd vote. Look at the second table to see the crowd forming. A single tree (1 estimator) starts at a shaky 0.814. By 10 trees the score has already climbed to about 0.906 and then it basically flattens out.

Going from 100 to 500 trees barely nudges the number while taking five times as long to run, so there is no reason to pay for the extra trees here. (The std (standard deviation) column wobbles a little run to run because these are tiny five-fold splits on a small toy dataset. On real, larger data the forest’s fold-to-fold scores settle into a tighter band than a single tree’s. The headline takeaway holds: the forest is both more accurate and more dependable than one tree.)

Feature Importance: Which Features Matter?

Once the forest is built, you usually want to know which columns actually drove the decisions. That is feature importance: a score for each feature saying how much it helped the trees separate the classes. There are two ways to measure it, and they do not always agree.

Picture a hiring panel. The built-in “impurity” importance is like asking each interviewer how often they brought up a topic. Someone who mentions “years of experience” in every interview looks influential just because they talk about it a lot, even if it rarely changes the decision. The second method, permutation importance, is fairer: it shuffles one feature into nonsense and checks how much the model’s accuracy actually drops. If scrambling a column barely hurts, that column was not really pulling its weight. When the two methods disagree, trust the permutation version.

📄 feature_importance.py: two ways to score each feature

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                            random_state=42)
feature_names = [f"feat_{i}" for i in range(10)]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# Method 1: Impurity-based importance (built-in, fast but biased)
print("Impurity-based importance (built-in):")
for name, imp in sorted(zip(feature_names, rf.feature_importances_), key=lambda x: -x[1])[:5]:
    print(f"  {name}: {imp:.4f}")

# Method 2: Permutation importance (unbiased, slower)
perm_imp = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
print(f"\nPermutation importance (unbiased):")
for name, imp, std in sorted(zip(feature_names, perm_imp.importances_mean,
                                   perm_imp.importances_std), key=lambda x: -x[1])[:5]:
    print(f"  {name}: {imp:.4f} ± {std:.4f}")

print(f"\nUse permutation importance for reliable results.")
print(f"Impurity importance can overvalue high-cardinality features.")

▶ Output

Impurity-based importance (built-in):
  feat_0: 0.2014
  feat_1: 0.1867
  feat_5: 0.1709
  feat_7: 0.0922
  feat_3: 0.0898

Permutation importance (unbiased):
  feat_5: 0.1560 ± 0.0332
  feat_1: 0.1070 ± 0.0168
  feat_0: 0.0490 ± 0.0212
  feat_7: 0.0390 ± 0.0170
  feat_8: 0.0250 ± 0.0180

Use permutation importance for reliable results.
Impurity importance can overvalue high-cardinality features.

What happened here: Both methods agree that feat_0, feat_1 and feat_5 are the heavy hitters, but they rank them differently. The impurity method puts feat_0 on top, while the permutation method, the one that actually measures lost accuracy, crowns feat_5 instead and nearly doubles its score over second place. That gap is the whole point. The built-in numbers come free with every fitted forest and are great for a quick glance, but they lean toward features with many distinct values.

The permutation numbers cost a few extra seconds of compute and tell you what really moves the needle. The little ± 0.0332 next to each one is the spread across the ten reshuffles, so a feature whose score is smaller than its own spread is basically noise. When you have to defend a model to a stakeholder, quote the permutation numbers.

Out-of-Bag Score: Free Validation

Here is a quiet bonus that Random Forest gives you for free. Remember that each tree trains on a bootstrap sample, a random draw of rows with replacement. Because of the way that draw works, roughly a third of the rows never make it into any given tree. Those left-out rows are called out-of-bag, and the tree has never seen them, so they make a perfect little test set for that tree.

Think of a potluck where each cook tastes only the dishes they did not bring. Every cook ends up scoring food they had no hand in making, so nobody is grading their own cooking. Add up all those honest tastings and you get a fair review of the whole spread, without setting aside a separate panel of judges. The out-of-bag (OOB) score works the same way: set oob_score=True and you get a validation estimate without ever carving out a hold-out set.

📄 oob_score.py: validation that costs you nothing extra

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score

X, y = make_classification(n_samples=500, n_features=10, random_state=42)

# oob_score=True gives you a validation score for free
rf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
rf.fit(X, y)

cv_score = cross_val_score(rf, X, y, cv=5).mean()

print(f"OOB Score:          {rf.oob_score_:.3f}")
print(f"5-Fold CV Score:    {cv_score:.3f}")
print(f"Difference:         {abs(rf.oob_score_ - cv_score):.3f}")
print(f"\nOOB approximates CV without needing to split data.")
print(f"Each tree tests on the ~37% of samples it did NOT train on.")

▶ Output

OOB Score:          0.946
5-Fold CV Score:    0.944
Difference:         0.002

OOB approximates CV without needing to split data.
Each tree tests on the ~37% of samples it did NOT train on.

What happened here: The OOB score (0.946) lands within 0.002 of the five-fold cross-validation score (0.944), and it came practically free. Cross-validation had to refit the forest five separate times; the OOB estimate fell out of a single fit using rows the trees had already skipped. That ~37% figure is not a guess. The chance a specific row dodges a bootstrap draw of size n is (1 – 1/n) to the power n, which settles near 1/e, about 0.368, as the dataset grows. On big datasets, where running full cross-validation gets expensive, OOB gives you a trustworthy accuracy read for the price of one fit.

Key Hyperparameters

Think of a new camera. On full auto it already takes sharp photos, and most people never touch a dial. A few manual settings exist for the moments you want more control, but the defaults carry you most of the way. Random Forest is like that. It is forgiving, which is exactly why it is a great default. You can leave almost everything alone and still get a strong model.

But four knobs are worth knowing. n_estimators is how many trees you grow (more is steadier, up to a point). max_depth caps how deep each tree can go. max_features sets how many features a tree may consider at each split, and it is the dial that controls how different the trees are from one another. min_samples_leaf sets the smallest number of rows allowed in a leaf, which keeps trees from memorizing single oddball rows.

The run below tries a few common setups on the same data so you can see how little they move the needle.

📄 hyperparameters.py: trying the four knobs that matter

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=10, random_state=42)

# n_estimators: more trees = better (until diminishing returns)
# max_depth: controls individual tree complexity
# max_features: number of features considered at each split
# min_samples_leaf: minimum samples in a leaf

configs = [
    ("Default", {}),
    ("More trees", {"n_estimators": 500}),
    ("Shallow trees", {"max_depth": 5}),
    ("Few features", {"max_features": 3}),
    ("Large leaves", {"min_samples_leaf": 10}),
    ("Tuned", {"n_estimators": 200, "max_depth": 8, "min_samples_leaf": 5}),
]

print(f"{'Config':<16} | {'CV Score':>10}")
print("-" * 30)
for name, params in configs:
    rf = RandomForestClassifier(random_state=42, **params)
    scores = cross_val_score(rf, X, y, cv=5)
    print(f"{name:<16} | {scores.mean():>9.3f}")

▶ Output

Config           |   CV Score
------------------------------
Default          |     0.944
More trees       |     0.948
Shallow trees    |     0.946
Few features     |     0.944
Large leaves     |     0.910
Tuned            |     0.942

What happened here: Notice how flat the column is. The plain default already scores 0.944, and the best variant (“More trees” at 0.948) beats it by a whisker. Throwing 500 trees at the problem buys you 0.004 for five times the compute. Capping depth or features barely changes anything. The one setup that actually hurt was “Large leaves” (min_samples_leaf=10), which dropped to 0.910 because forcing at least ten rows into every leaf made the trees too blunt to catch the finer patterns.

And here is the honest part most tutorials skip: the hand-“Tuned” combo (0.942) came in just below the untouched default. On a clean dataset like this, fiddling can make things slightly worse, not better. The real lesson is to start with the defaults, change one knob at a time, and only keep a change if cross-validation clearly rewards it.

Common Mistakes

Mistake 1: Piling on trees and burning CPU for nothing

Beginners often assume that if 100 trees are good, 10,000 must be ten times better. They are not. Adding trees never hurts accuracy, but the gains flatten out fast, and you pay for every extra tree in training time and prediction time. It is like inviting more people to vote on a decision the crowd already agreed on hours ago.

❌ Mistake: using too many trees and wasting computation

# n_estimators=10000 gives barely better accuracy than n_estimators=200,
# but takes far longer to train and predict.
# Plot accuracy vs n_estimators and you will see a plateau around 100 to 300.
# Beyond that, you are spending CPU for a 0.001 improvement.

# Good starting point: n_estimators=100 to 300
# Increase only if cross-validation is still climbing
print("Random Forest free lunch: more trees never hurt accuracy.")
print("But past 200 to 300 trees, the improvement is negligible.")
print("Cost grows linearly: doubling trees roughly doubles train and predict time.")

▶ Output

Random Forest free lunch: more trees never hurt accuracy.
But past 200 to 300 trees, the improvement is negligible.
Cost grows linearly: doubling trees roughly doubles train and predict time.

Mistake 2: Assuming Random Forest still cannot read missing values

For years the standard advice was “scikit-learn’s Random Forest cannot handle NaN, so impute first”. That advice is now out of date. Since the missing-value support added to scikit-learn’s trees (1.3 for single trees, 1.4 for the forest), and confirmed here on scikit-learn 1.9.0, RandomForestClassifier fits and predicts with NaN values straight out of the box. The tree learns, at each split, which direction missing rows should travel. You may still choose to impute for other reasons, but you are no longer forced to.

✅ Verified on scikit-learn 1.9.0: NaN trains without a separate imputer

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=10, n_informative=5,
                            random_state=42)

# Punch some holes in the data on purpose
rng = np.random.RandomState(0)
X[rng.choice(500, 50, replace=False), 0] = np.nan
print("Missing values present:", int(np.isnan(X).sum()))

# No SimpleImputer needed: the forest handles NaN itself
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)
print("Fit + predict with NaN worked. First 5 predictions:", rf.predict(X[:5]))

▶ Output

Missing values present: 50
Fit + predict with NaN worked. First 5 predictions: [0 0 0 0 1]

Practice Exercises

1. Exercise 1: Load the built-in load_wine dataset and compare a single DecisionTreeClassifier against a 100-tree RandomForestClassifier using five-fold cross-validation. How big is the gap on real data?
2. Exercise 2: Fit a forest, then compute both impurity importance and permutation importance for every feature. Sort them and see where the two rankings disagree.
3. Exercise 3: Run GridSearchCV over n_estimators, max_depth, and max_features. Then check whether the “best” combo actually beats the plain defaults, the way the Tuned row did not in our table above.

Conclusion

You now know why a crowd of “good enough” trees beats one clever tree: bootstrap samples and random feature picks force the trees to disagree, so their mistakes cancel and the real signal comes through on the majority vote. You saw a single tree at about 86% jump to 91% just by voting, feature importance measured two ways (and why permutation is the one to trust), the out-of-bag score that hands you free validation, and the handful of hyperparameters that barely move the needle on clean data. The big takeaway: start with the defaults, change one knob at a time, and only keep a change when cross-validation clearly rewards it.

Next up is Gradient Boosting with XGBoost, LightGBM, and CatBoost, the family that trades Random Forest’s easy tuning for a shot at the last point or two of accuracy. For the full path from Python basics through machine learning, head to the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Random Forest

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: What are the two sources of randomness in a Random Forest, and why do both matter?

First, each tree trains on a bootstrap sample: a random draw of rows taken with replacement, so every tree sees a slightly different dataset. Second, at each split a tree may only consider a random subset of features (max_features), not all of them. Bagging alone would still leave the trees highly correlated because one or two strong features would dominate every tree’s top splits. The random feature subset breaks that correlation, so the trees make different mistakes and their errors cancel when you average or vote.

Q: Why is permutation importance usually more trustworthy than the built-in impurity importance?

Impurity importance counts how often and how much a feature reduced impurity across all splits, which biases it toward high-cardinality features (many distinct values) and features used early, even when they do not truly drive accuracy. Permutation importance instead shuffles one feature into noise on held-out data and measures how much the model’s score drops, so it reflects real predictive contribution. When the two disagree, quote the permutation numbers, especially when defending a model to a stakeholder.

Q: What is the out-of-bag score and where does the roughly 37% figure come from?

Because each tree trains on a bootstrap sample drawn with replacement, some rows never make it into a given tree. Those out-of-bag rows act as a free test set for that tree, and averaging over all trees gives an OOB score that closely approximates cross-validation from a single fit. The probability a specific row is missed is (1 – 1/n) to the power n, which approaches 1/e (about 0.368) as n grows, so roughly 37% of rows are out-of-bag for each tree.

Q: Does adding more trees ever cause a Random Forest to overfit?

No. Unlike boosting, adding trees to a Random Forest does not increase overfitting, because each tree is trained independently and the ensemble simply averages more votes. The score climbs, then plateaus, usually around 100 to 300 trees on typical data. Past that point you spend linearly more training and prediction time for a negligible accuracy gain, so more trees waste Central Processing Unit (CPU) rather than hurt accuracy.

Q: Scenario: your Random Forest hits 99% accuracy on training data but only 78% on the test set. What do you check first?

That gap points to overfitting, so first look at per-tree complexity rather than tree count. Cap max_depth and raise min_samples_leaf so individual trees stop memorizing single oddball rows, and lower max_features to decorrelate the trees further. Also confirm there is no data leakage (a feature that secretly encodes the target) and that the train/test split is clean, since near-perfect training accuracy with a poor test score is a classic leakage signature. Use the OOB or cross-validation score, not the training score, to judge each change.

Q: Scenario: a teammate feeds a dataset with NaN values into RandomForestClassifier and expects a crash, but it trains fine. Why?

Modern scikit-learn added native missing-value support to trees in 1.3 and to the forest in 1.4, and it works on 1.9.0. At each split the tree learns which direction missing rows should travel, so RandomForestClassifier fits and predicts with NaN directly, no separate imputer required. The old “you must impute first” rule is out of date, though you may still choose to impute for other reasons such as sharing the pipeline with models that cannot handle NaN.

Q: How does predict_proba work in a Random Forest, and when should you not trust it directly?

predict_proba() returns the share of trees voting for each class, so if 73 of 100 trees pick class 1, it reports 0.73 for class 1. These vote-share probabilities are reasonable for ranking and thresholding, but they are not guaranteed to be well calibrated, meaning a reported 0.73 does not always match a true 73% chance. For tasks that depend on exact probabilities (risk scoring, expected-value decisions), wrap the model in CalibratedClassifierCV to fix the calibration.

Series: Python + AI/ML Cookbook, Part 5: Machine Learning

Further reading: for the full reference, see scikit-learn documentation.

Related Posts

Previous: ML: Decision Trees in Python: Splitting, Pruning, Visualization

Next: ML: Gradient Boosting with XGBoost, LightGBM, CatBoost

Series Home: Python + AI/ML Tutorial Series