Model evaluation metrics are the numbers that tell you whether your machine learning model actually works. This guide covers accuracy and the trap it hides, the precision versus recall tradeoff, the F1 score that balances the two, and the bias-variance idea that sits underneath all of it.
“A model is only as good as the metric you judge it by. Pick the wrong metric, and you happily optimize for the wrong thing.”
Andriy Burkov, The Hundred-Page Machine Learning Book
Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Intermediate | Reading Time: 12 minutes
Accuracy is the first model evaluation metric everyone learns, and the one that lies the most. Picture a quiet building with a security guard who has never seen a real break-in. He could “predict” no burglar every single night and be right 99 percent of the time, while catching exactly zero burglars. That is accuracy on imbalanced data in one sentence. If 95 percent of emails are legitimate, a model that says “not spam” for everything scores 95 percent accuracy and catches no spam at all. In fraud detection, medical screening, and defect inspection, a missed positive costs far more than a false alarm, so accuracy alone is dangerous.
This post walks through when accuracy is fine (balanced data, equal costs), when precision matters (false alarms are expensive), when recall matters (missed positives are dangerous), and how the F1 score balances both when you cannot give up either one. Every number you see below came from running the code on Python 3.14.6 with scikit-learn 1.9.0, so you can reproduce it exactly.
Table of Contents
Prerequisites
The Metrics Cheat Sheet
Most people land here from a search wanting one quick answer: which metric do I use, and what does it mean? Here is the whole table up front. Read it once, then the rest of the post explains the rows that trip people up.
| Metric | In plain words | Formula | Use it when |
|---|---|---|---|
| Accuracy | Fraction of all predictions that were right | (TP + TN) / total | Classes are balanced and a miss costs the same as a false alarm |
| Precision | Of the things you flagged, how many were truly positive | TP / (TP + FP) | A false alarm is expensive (spam filter, marketing email) |
| Recall | Of the real positives, how many you actually caught | TP / (TP + FN) | A miss is dangerous (cancer screening, fraud) |
| F1 score | A single number that balances precision and recall | 2 · P · R / (P + R) | Classes are imbalanced and you need both reasonably high |
TP, TN, FP, and FN stand for true positive, true negative, false positive, and false negative. Here is a quick way to keep precision and recall straight. Think of a fishing net. Recall asks: of all the fish in the lake, how many did my net pull in? Precision asks: of everything in my net, how much was actual fish and not old boots and weeds? A huge net catches every fish (high recall) but scoops up junk too (low precision). A tiny careful net brings up only fish (high precision) but lets most of them swim away (low recall). F1 is the score that rewards a net good at both.
The Bias-Variance Tradeoff
Before the metrics, one idea explains why models go wrong in the first place. Imagine a student studying for an exam. One student barely reads the book and answers everything with the same lazy guess. That is high bias: too simple, misses the real pattern, this is underfitting. Another student memorizes every word on every page, including the typos, then freezes on a question worded slightly differently. That is high variance: too tied to the exact training examples, this is overfitting. The student you want sits in the middle: learned the real ideas, can handle new questions.
Every model sits somewhere on this spectrum. A decision stump (a tree with a single split) underfits: high bias, it misses the shape of the data. A 1000-node decision tree with no depth limit overfits: it memorizes the noise in the training set. Cross-validation is how you find out where your model actually sits. If your training accuracy is 99 percent but your validation accuracy is 70 percent, that gap is the tell: you are overfitting, and the model learned the textbook typos instead of the ideas.
The Accuracy Trap
Let us prove the security-guard story with real code. We build a fraud dataset where only 5 percent of transactions are fraud, then compare two “models”: a lazy one that always predicts “not fraud”, and a real logistic regression. Watch what accuracy alone hides.
📄 accuracy_trap.py: why accuracy lies on imbalanced data
import numpy as np
from sklearn.metrics import accuracy_score, classification_report
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
# 95% class 0, 5% class 1 (a fraud-detection scenario)
X, y = make_classification(n_samples=4000, n_features=10,
weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Dummy "model": always predict the majority class (no fraud, ever)
dummy = np.zeros_like(y_test)
# Real model
model = LogisticRegression(random_state=42)
model.fit(X_train, y_train)
preds = model.predict(X_test)
print(f"Dummy accuracy: {accuracy_score(y_test, dummy):.1%} (catches 0 fraud)")
print(f"Model accuracy: {accuracy_score(y_test, preds):.1%}")
print(f"\n{classification_report(y_test, preds, target_names=['Legit', 'Fraud'])}")
▶ Output
Dummy accuracy: 95.0% (catches 0 fraud)
Model accuracy: 98.6%
precision recall f1-score support
Legit 0.99 1.00 0.99 1140
Fraud 0.98 0.73 0.84 60
accuracy 0.99 1200
macro avg 0.98 0.87 0.92 1200
weighted avg 0.99 0.99 0.98 1200
What happened here: The dummy that does literally nothing scores 95.0 percent accuracy, because 95 percent of the test set is legit and it always guesses legit. It catches zero fraud. The classification report is what saves you: look at the Fraud row. The real model has 0.73 recall, so it catches 73 percent of the actual fraud, with 0.98 precision, meaning almost every fraud flag is correct. The single accuracy number could never tell you that. For any imbalanced problem, always read the per-class precision, recall, and F1 instead of trusting one accuracy figure. Your exact numbers will match these because we seeded everything with random_state=42.
Handling Imbalanced Classes
Spotting the accuracy trap is half the job. The other half is fixing it, so the model actually learns the rare class instead of shrugging it off. You have three levers, and they stack: you can reweight the loss, reshape the data, or retune the cutoff. Start with the cheapest one that solves your problem and only add the next lever if recall is still too low.
Class weighting is the one-line fix, and it is usually where you should start. You tell the model to treat the rare class as more important. In scikit-learn, pass class_weight='balanced' to a classifier like LogisticRegression or RandomForestClassifier, and it scales each class’s contribution to the loss inversely to how often that class shows up. A fraud case that appears 5 percent of the time then counts for roughly twenty times as much as a legit one, so a wrong call on fraud stings the model far more during training and it stops quietly ignoring the minority. Nothing about your data changes, only how much each mistake costs. When the imbalance is mild to moderate and you just want the model to stop favoring the majority, this is the first thing to try.
Resampling changes the training data itself instead of the loss. You can push in two directions. Oversampling grows the minority, and the popular method is SMOTE (Synthetic Minority Over-sampling Technique), which does not just copy rare rows but invents new synthetic ones by interpolating between a real minority example and its nearest neighbors. That gives the model a fuller picture of what fraud looks like without handing it exact duplicates it can memorize. Undersampling goes the other way and drops some of the plentiful majority rows until the classes sit closer to even, which is faster and lighter but risks throwing away useful signal. Both live in the imbalanced-learn library, which plugs into the same scikit-learn workflow you already use. Reach for resampling when class weighting alone did not move recall enough, and prefer SMOTE when you have enough real minority examples to interpolate between sensibly. One rule you cannot skip: resample only the training split, never the test set, or you will fool yourself with leaked data and numbers that fall apart in production.
Decision-threshold tuning is the cheapest lever of all, because it needs no retraining. As the threshold section below shows, a classifier really outputs a probability, and the default cutoff is 0.5. Nothing forces you to keep it there. Slide the cutoff down toward 0.3 and the model flags more cases, trading precision for recall so fewer real positives slip past. Slide it up toward 0.7 and it gets choosier, buying precision at the cost of recall. This is the lever to reach for when the model already ranks cases well (its probabilities are trustworthy) and you simply need to match the cutoff to what a miss versus a false alarm actually costs your business.
A quick way to decide which one to reach for:
- Class weighting first. One argument, no new libraries, no change to your data. Best for mild to moderate imbalance.
- Resampling with SMOTE or undersampling when weighting is not enough and the imbalance is severe. Oversample when minority data is scarce, undersample when the majority is huge and you have compute to spare.
- Threshold tuning any time, often on top of the other two, to set the final precision versus recall balance to your real-world costs.
Precision vs Recall: The Threshold Tradeoff
A classifier does not really output “fraud” or “not fraud”. It outputs a probability, like 0.62. You then pick a cutoff, the threshold, above which you call it fraud. Move that cutoff and you trade precision for recall, the same way tightening or loosening that fishing net changes what you bring back. Here is the dial in action.
📄 threshold_tradeoff.py: how the cutoff moves precision and recall
from sklearn.metrics import precision_score, recall_score, f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
# Overlapping classes so the threshold genuinely changes the outcome
X, y = make_classification(n_samples=4000, n_features=10, n_informative=5,
weights=[0.8, 0.2], class_sep=1.0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
model = LogisticRegression(random_state=42).fit(X_train, y_train)
probas = model.predict_proba(X_test)[:, 1]
print("Threshold | Precision | Recall | F1")
print("----------|-----------|--------|------")
for t in [0.2, 0.3, 0.4, 0.5, 0.6, 0.7]:
p_hat = (probas >= t).astype(int)
p = precision_score(y_test, p_hat, zero_division=0)
r = recall_score(y_test, p_hat, zero_division=0)
f = f1_score(y_test, p_hat, zero_division=0)
print(f" {t:.1f} | {p:>8.3f} | {r:>5.3f} | {f:>5.3f}")
▶ Output
Threshold | Precision | Recall | F1 ----------|-----------|--------|------ 0.2 | 0.487 | 0.823 | 0.612 0.3 | 0.603 | 0.759 | 0.672 0.4 | 0.718 | 0.647 | 0.680 0.5 | 0.770 | 0.547 | 0.640 0.6 | 0.795 | 0.384 | 0.517 0.7 | 0.792 | 0.263 | 0.395
What happened here: Drop the threshold to 0.2 and the model flags almost everything: recall jumps to 0.82 (it catches most of the real positives) but precision sags to 0.49 (about half of its flags are false alarms). Raise the threshold to 0.7 and it gets picky: precision climbs to 0.79 but recall falls to 0.26, so it now misses three out of four real positives. F1, the balance of the two, peaks at threshold 0.4 (0.680). There is no universally correct cutoff. Your problem decides: cancer screening leans toward high recall (never miss a case), email marketing leans toward high precision (do not annoy real customers). The threshold is a business decision wearing a math costume.
Common Mistakes
Mistake 1: Grading the model on the data it studied
This is the most common beginner trap. You let a deep decision tree memorize the training set, then measure it on that same set and celebrate. It is like giving a student the exact exam questions to study, then being amazed they score 100 percent. The real test is data the model has never seen.
❌ Mistake: evaluating on training data
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=500, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
dt = DecisionTreeClassifier(random_state=42) # no depth limit, it will memorize
dt.fit(X_train, y_train)
print(f"Training accuracy: {accuracy_score(y_train, dt.predict(X_train)):.1%}")
print(f"Test accuracy: {accuracy_score(y_test, dt.predict(X_test)):.1%}")
print("The gap is the overfitting. Always score on held-out data.")
▶ Output
Training accuracy: 100.0% Test accuracy: 90.7% The gap is the overfitting. Always score on held-out data.
What happened here: The tree nailed 100 percent on data it had already seen and 90.7 percent on fresh data. That 9-point gap is overfitting you can measure. A perfect training score is a warning sign, not a trophy. Always report the number on held-out test data, never on the data the model trained on.
Mistake 2: Reporting accuracy on imbalanced data
If one class is rare, accuracy will look great while the model quietly ignores the class you care about, exactly like the dummy fraud model above. The fix is simple: report precision, recall, and F1 for the rare class, and pick the metric that matches your cost. A missed fraud is expensive, so you watch recall. Use classification_report and read the minority row, not the headline accuracy.
Conclusion
You now know why accuracy lies on imbalanced data, how precision and recall pull against each other, why the F1 score balances them, and how bias and variance explain the whole thing underneath. The one habit to carry forward in model evaluation: never trust a single number. Read the per-class precision, recall, and F1, then pick the metric that matches the cost of a mistake in your problem. Next up we turn these numbers into pictures with the confusion matrix and Receiver Operating Characteristic (ROC) curves, which make the same tradeoffs even easier to see at a glance. For the full path from Python basics to deployed models, see the Python + AI/ML tutorial series home.
Frequently Asked Questions
What are model evaluation metrics?
Model evaluation metrics are numbers that measure how well a machine learning model performs. The core classification metrics are accuracy (fraction of correct predictions), precision (how many flagged positives were real), recall (how many real positives were caught), and F1 (the balance of precision and recall). You pick the metric that matches the cost of a mistake in your problem.
When should I use F1 instead of accuracy?
Use accuracy when classes are balanced and a false positive costs about the same as a false negative. Use F1 when classes are imbalanced or when you need precision and recall to both stay reasonably high. On a 95/5 split, accuracy can read 95 percent while the model catches none of the rare class, so F1 is the safer choice there.
What is the difference between macro and weighted F1?
Macro F1 averages the F1 of every class with equal weight, regardless of how many samples each class has. Weighted F1 averages them weighted by class size (support). Use macro when every class matters equally, even the rare ones. Use weighted when you want a score that reflects overall performance on the data as it actually is distributed.
How do I fix a model with low recall?
Lower the decision threshold so the model flags more positives, set class_weight to penalize missed positives, balance the data with techniques like SMOTE (Synthetic Minority Over-sampling Technique), or collect more examples of the positive class. Lowering the threshold is the fastest lever: it raises recall at the cost of some precision.
Can I optimize for a specific metric during training?
Yes. Pass the metric name to the scoring parameter in cross-validation, for example cross_val_score(model, X, y, scoring=’f1′). For imbalanced problems, scoring=’f1_weighted’ or scoring=’roc_auc’ is usually a better target than the default accuracy.
Interview Questions on Model Evaluation
The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.
Q: Explain precision and recall in one sentence each, without the formulas.
Precision answers “of the items I flagged as positive, how many were actually positive,” so it measures how trustworthy your alarms are. Recall answers “of all the real positives out there, how many did I actually catch,” so it measures how complete your coverage is. Precision punishes false alarms, recall punishes misses.
Q: Why is the F1 score a harmonic mean of precision and recall instead of a plain average?
The harmonic mean stays low unless both numbers are high, so it refuses to reward a lopsided model. A model with 1.0 precision and 0.0 recall has a plain average of 0.5 but an F1 of 0.0, which correctly signals it is useless. That is why F1 is the go-to single number for imbalanced problems where you cannot let either metric collapse.
Q: Your fraud model reports 99 percent accuracy in a demo, but the business team says fraudulent transactions are still slipping through. What do you check first?
Check the class balance and then the per-class recall for the fraud class, not the headline accuracy. On a heavily imbalanced dataset a model can hit 99 percent accuracy by mostly predicting “not fraud” while catching very few real fraud cases. Run classification_report, read the fraud row’s recall, and if it is low, lower the decision threshold or set class_weight="balanced".
Q: A product manager wants your spam filter to stop sending real emails to the spam folder, even if a little spam gets through. Which metric do you optimize and which way do you move the threshold?
Optimize precision, because a real email marked as spam is a false positive and that is the cost they want to avoid. Raise the decision threshold so the model only calls something spam when it is very confident. The tradeoff is lower recall, meaning more spam slips into the inbox, which is the compromise the product manager explicitly accepted.
Q: You see 100 percent training accuracy and 78 percent test accuracy. What is happening and how do you reduce the gap?
That large gap is overfitting: the model memorized the training data, including its noise, and does not generalize. In terms of bias and variance, this is low bias and high variance. Reduce it by simplifying the model (limit tree depth, add regularization), gathering more training data, or using cross-validation to catch the problem earlier.
Q: Can a model have high precision and low recall at the same time? What does that model behave like?
Yes, and it is common when you set a high threshold. The model only predicts positive when it is very sure, so almost every flag is correct (high precision), but it stays silent on the borderline cases and misses many real positives (low recall). It behaves like an overly cautious inspector who is right whenever they speak up but lets most defects pass unchecked.
Series: Python + AI/ML Cookbook, Part 5: Machine Learning
Reference: the complete, always-current details live in scikit-learn documentation.
Related Posts
Previous: ML: Logistic Regression, Binary and Multi-class
Next: ML: Confusion Matrix & ROC Curves
Series Home: Python + AI/ML Tutorial Series
No comment