These ML best practices cover the ethics and the catches that nobody warns you about: how to spot bias in your data, how to measure fairness across different groups of people, how to explain what your machine learning model is actually doing, and the responsible AI checklist that separates a production-grade model from a dangerous prototype.

Fairness in machine learning is not a purely technical problem. That is the core lesson of the Gender Shades study.

after Joy Buolamwini & Timnit Gebru, Gender Shades (2018)

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Advanced | Reading Time: 16 minutes

A model is a mirror. It reflects back whatever you show it, only louder. Show it ten years of hiring decisions where mostly men got the job, and it learns one rule: prefer men. It will not feel guilty about it. It will just keep applying that rule, thousands of times a second, to every resume that lands in the queue. Amazon hit exactly this wall (revealed by Reuters in 2018, scrapped earlier) with an internal recruiting tool that had taught itself to downrank resumes containing the word “women”.

Here is the everyday version. Imagine teaching a kid to cook using only your family recipes. The kid grows up convinced that every meal needs garlic, because every recipe you handed over had garlic in it. The kid is not wrong about your kitchen. The kid is just blind to every kitchen that does things differently. A model trained on biased data is that kid: confident, fast, and quietly repeating the one pattern it was fed. Your job is to catch that pattern before it ships.

So this post is not an ethics lecture. It is a set of checks you run in code. Bias detection is a few lines of pandas, not a philosophy debate. Fairness is a number you compute and compare, not a value you sit and ponder. Explainability is a function call. If your model touches real people, these checks are as routine as writing a test, and we will run every one of them on Python 3.14.6 with scikit-learn 1.9.0.

Prerequisites

The Responsible AI Checklist

Read the diagram top to bottom. Once you have built a model, three ML best practices stand between you and a responsible deployment. First, a data bias audit: is every group represented well enough to learn from? Second, fairness metrics: do the outcomes look roughly even across groups, or is one group quietly getting a worse deal? Third, explainability: can you say, in plain words, why the model made each call? If you cannot answer all three with a confident yes, you are not ready to ship. Think of it as the pre-flight checklist a pilot runs before takeoff. Skipping it does not mean the plane crashes today, it means you have no idea when it will.

Detecting Bias in Data

Imagine two students competing for the same scholarship, but one is graded on a much harsher scale and only a handful of their answer sheets even make it to the judges’ table. No matter how sharp that student is, the deck is stacked against them. Bias in data works exactly like that. It hides in two places: in who shows up in your data, and in how the target was decided for them.

The first check is representation. If group B is only a third of your rows, the model simply has fewer examples to learn from, so it guesses worse for that group. The second check is the outcome itself. Even with perfect representation, if group B was historically held to a tougher standard, that unfairness is baked into the labels you are training on. Let us build a small loan-approval dataset that has both problems on purpose, then catch them.

📄 bias_detection.py: checking for representation imbalance

import numpy as np
import pandas as pd
from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

# Simulated loan approval data
rng = np.random.default_rng(42)
n = 1000

df = pd.DataFrame({
    "income": rng.normal(50000, 15000, n),
    "credit_score": rng.normal(700, 50, n),
    "age": rng.integers(22, 34, n),
    "group": rng.choice(["A", "B"], n, p=[0.7, 0.3]),  # imbalanced on purpose
})

# Biased target: group B is held to a harder threshold
df["approved"] = 0
mask_a = (df["group"] == "A") & (df["credit_score"] > 680)
mask_b = (df["group"] == "B") & (df["credit_score"] > 720)  # higher bar!
df.loc[mask_a, "approved"] = 1
df.loc[mask_b, "approved"] = 1

# Check 1: representation. Does every group have enough rows?
print("Group representation:")
print(df["group"].value_counts())
print()
print("Approval rates by group:")
approval_rates = df.groupby("group")["approved"].mean()
for group, rate in approval_rates.items():
    print(f"  Group {group}: {rate:.1%} approved")

# Check 2: per-group model performance. Scale first so the model converges.
X = df[["income", "credit_score", "age"]]
y = df["approved"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)
groups_test = df.loc[X_test.index, "group"]

model = make_pipeline(
    StandardScaler(),
    LogisticRegression(random_state=42),
).fit(X_train, y_train)
preds = model.predict(X_test)

print("\nPer-group model performance:")
for group in ["A", "B"]:
    mask = groups_test == group
    report = classification_report(
        y_test[mask], preds[mask], output_dict=True, zero_division=0)
    print(f"  Group {group}: precision={report['1']['precision']:.2f}, "
          f"recall={report['1']['recall']:.2f}")

▶ Output

Group representation:
group
A    706
B    294
Name: count, dtype: int64

Approval rates by group:
  Group A: 63.6% approved
  Group B: 32.3% approved

Per-group model performance:
  Group A: precision=1.00, recall=0.83
  Group B: precision=0.60, recall=1.00

What happened here: Both checks fired. Representation is lopsided (706 rows for group A, 294 for group B), and the approval rates are nowhere near each other (63.6% for A, 32.3% for B). That gap alone is a red flag, because we built the data so group B faced a tougher credit threshold, and the model dutifully soaked that up. Now look at the per-group scores. They are not just lower for one group, they are differently shaped.

For group A the model is never wrong when it says “approve” (precision 1.00) but it misses some good applicants (recall 0.83). For group B it catches every good applicant (recall 1.00) but four out of ten of its approvals are mistakes (precision 0.60). Same model, two very different experiences depending on which group you land in. That mismatch, not the raw accuracy, is the thing to chase. And the fix starts in the data, not the model: no amount of tuning undoes a label that was unfair to begin with.

Model Explainability with SHAP

A model that says “denied” without saying why is a model you cannot defend, to a regulator or to the person you just denied. Explainability is how you make it talk. The headline tool here is SHAP (SHapley Additive exPlanations), which borrows an idea from game theory: treat each feature like a player on a team and work out how much credit each one deserves for a given prediction. SHAP is a separate install (pip install shap), and its real strength is per-prediction explanations, telling you why this one applicant got denied, not just which features matter on average.

Before reaching for SHAP, there is a humbler check worth running first, and it ships with scikit-learn. Every tree-based model hands you a built-in feature_importances_ list, and it is tempting to trust it. The problem is that built-in importance is easily fooled: it rewards features just for having lots of distinct values to split on, even when those values are pure noise. Permutation importance is the honest alternative. The idea is dead simple.

Shuffle one column so it becomes garbage, then see how much the model’s accuracy drops. A big drop means the model really leaned on that feature. No drop means the feature was never pulling its weight. Here is the contrast on a dataset where we know the truth in advance.

📄 feature_importance.py: built-in importance versus permutation importance

import numpy as np
from sklearn.inspection import permutation_importance
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Build data where we KNOW which features matter.
# income and credit_score drive approval; zipcode is pure noise.
rng = np.random.default_rng(42)
n = 600
income = rng.normal(50000, 15000, n)
credit_score = rng.normal(700, 50, n)
debt_ratio = rng.uniform(0, 1, n)
age = rng.integers(22, 34, n)
zipcode = rng.integers(10000, 99999, n)  # random, tells you nothing

# Approval depends only on the first three, plus a little noise
score = (
    (income - 50000) / 15000
    + (credit_score - 700) / 50
    - debt_ratio
    + rng.normal(0, 0.3, n)
)
y = (score > 0).astype(int)

X = np.column_stack([income, credit_score, debt_ratio, age, zipcode])
feature_names = ["income", "credit_score", "debt_ratio", "age", "zipcode"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42).fit(
    X_train, y_train)

# Built-in importance: fast, but easily fooled
print("Feature Importance (built-in):")
for name, imp in sorted(zip(feature_names, model.feature_importances_),
                        key=lambda x: -x[1]):
    print(f"  {name:<15} {imp:.3f}")

# Permutation importance: shuffle each column, measure the accuracy drop
perm = permutation_importance(
    model, X_test, y_test, n_repeats=10, random_state=42)
print("\nPermutation Importance (more reliable):")
for name, imp, std in sorted(
        zip(feature_names, perm.importances_mean, perm.importances_std),
        key=lambda x: -x[1]):
    print(f"  {name:<15} {imp:.3f} +/- {std:.3f}")

▶ Output

Feature Importance (built-in):
  income          0.383
  credit_score    0.372
  debt_ratio      0.134
  zipcode         0.072
  age             0.038

Permutation Importance (more reliable):
  credit_score    0.198 +/- 0.022
  income          0.193 +/- 0.038
  age             0.005 +/- 0.006
  debt_ratio      -0.001 +/- 0.014
  zipcode         -0.004 +/- 0.007

What happened here: We rigged the data so only income, credit score, and debt ratio decide the outcome. Age and zipcode are noise. Now compare the two lists. Built-in importance still hands zipcode a score of 0.072, more than age, even though zipcode is a random five-digit number that means nothing. Why? A random number has tons of unique values, so the tree finds plenty of places to split on it and racks up fake credit.

Permutation importance is not fooled. Shuffle credit_score and accuracy falls 0.198; shuffle income and it falls 0.193; those two clearly carry the model. Shuffle zipcode and accuracy barely moves (-0.004, basically zero, the tiny negative just means shuffling noise sometimes helps by luck). That is the tell: a feature you can scramble with no effect was never really being used. Reach for permutation importance or SHAP, and treat the built-in list as a rough first guess, not the final word.

Common Mistakes

Mistake 1: Deleting the sensitive column and calling it fair

This is the trap almost everyone falls into first. The logic feels airtight: if the model never sees "gender" or "race", how could it possibly discriminate on them? But the model does not need the column. It reconstructs it. Zip code stands in for neighborhood, neighborhood stands in for race. First name hints at gender. The school you attended, your shopping history, even your phone's operating system can leak the very thing you tried to hide. The model quietly rebuilds the forbidden feature from the leftovers and discriminates anyway. This is called proxy discrimination, and it is why deleting a column is necessary but nowhere near sufficient.

❌ The mistake: drop the column, assume the bias left with it

# Dropping "gender" or "race" from the features does NOT remove bias.
# Other columns (zipcode, name, school) quietly act as stand-ins.
# A model that never sees "race" can still discriminate through them.
# The name for this is proxy discrimination.
print("Removing the sensitive column is necessary but not sufficient.")
print("Audit predictions across groups anyway.")
print("If group A is approved 80% of the time and group B 50%,")
print("you have disparate impact, even without the column.")

▶ Output

Removing the sensitive column is necessary but not sufficient.
Audit predictions across groups anyway.
If group A is approved 80% of the time and group B 50%,
you have disparate impact, even without the column.

The fix: keep auditing outcomes by group even after you drop the sensitive column. The only honest test of fairness is the prediction distribution itself, not the feature list. If group A and group B walk away with very different approval rates, you have a problem, no matter which columns the model was allowed to look at.

Mistake 2: Trusting built-in feature importance

We already saw it bite: built-in feature_importances_ handed a random zip code more credit than a feature that genuinely mattered. Treat that list as a quick sanity check, never as evidence. When you need to explain a prediction to a colleague, an auditor, or the person it affected, use permutation importance for the global picture and SHAP for the per-prediction breakdown. Both are a few lines of code, and both will save you from confidently pointing at the wrong feature.

When You Will Reach For This

These ML best practices are not academic. They show up the moment your model starts making decisions about people, and that happens more often than you think.

• Anything that approves or denies a person. Loans, insurance quotes, apartment applications, job-screening scores. The instant the output changes someone's life, run the per-group audit from the bias section before it ever sees a real applicant.
• Anything a regulator can ask you to explain. Finance and healthcare increasingly require a reason for every automated decision. "The neural network said so" is not an answer. Permutation importance and SHAP are how you produce one.
• Internal models you think are low stakes. An employee-attrition or promotion-ranking model still decides who gets opportunities and who gets overlooked. Bias here is quieter, but the legal and trust damage is just as real.

Try It Yourself

Take the bias_detection.py script from above and extend it. Right now you have spotted the unfairness. Your job is to measure one fix and see whether it actually helps.

1. Compute the disparity directly. Print the difference in approval rate between group A and group B as a single number (the "demographic parity gap"). Anything far from zero is a warning.
2. Try to shrink it. Pass class_weight="balanced" to the LogisticRegression, retrain, and recompute the per-group precision and recall. Did the gap shrink, grow, or just move around?
3. Be honest about the catch. Write one sentence on what that fix cost you. Fairness adjustments almost always trade a little overall accuracy for a fairer split, and the right trade depends on who the model affects.

Conclusion

You started with a vague worry, "is my model fair?", and turned it into three checks you can actually run in code: audit representation and outcomes in the data, compare performance group by group, and explain every prediction with permutation importance or SHAP. You also saw the two traps that catch most people first: dropping a sensitive column and calling it fair, and trusting the built-in feature importance list. None of this is philosophy. These ML best practices boil down to a handful of pandas and scikit-learn calls you run before you ship, every single time your model touches a real person.

Next, you put the whole toolkit to work on a real dataset in the end-to-end house price prediction project. For the full roadmap from beginner basics all the way to production ML, head to the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on ML Best Practices

Interviewers rarely ask for definitions. They ask what happens in situations like these.

Q: What is proxy discrimination, and why does removing a sensitive feature not guarantee a fair model?

Proxy discrimination happens when the model rebuilds a forbidden attribute (like gender or race) from other columns that correlate with it, such as zip code, first name, or the school you attended. Because the model reconstructs the signal indirectly, dropping the sensitive column removes it from view but not from influence. That is why removing the column is necessary but not sufficient: you still have to audit the prediction distribution across groups to confirm outcomes are actually even.

Q: Why is permutation importance more trustworthy than the built-in feature_importances_ of a tree model?

Built-in importance rewards a feature just for having many distinct values to split on, so a high-cardinality noise column like a random zip code can score higher than a feature that genuinely matters. Permutation importance instead shuffles one column at a time and measures how much accuracy drops. A feature you can scramble with no effect on accuracy was never really being used, which makes permutation importance a much more honest signal of what the model actually leans on.

Q: A compliance officer asks why your model denied one specific applicant. Which tool do you reach for, and why?

Reach for SHAP, because its strength is per-prediction explanations: it assigns each feature a signed contribution to that one applicant's score, so you can say exactly which factors pushed the decision toward denial. Global tools like permutation importance only tell you which features matter on average across everyone, which does not answer a question about a single person. SHAP's Shapley-value foundation also gives it stronger consistency guarantees than a quicker method like LIME.

Q: Your loan model reports 92% overall accuracy, but the risk team is still worried. You split accuracy by group and find recall is 0.95 for group A and 0.60 for group B. What do you conclude and do next?

Overall accuracy is hiding a serious fairness gap: the model is catching almost all qualified group A applicants but missing 40% of qualified group B applicants, so group B is systematically underserved. The aggregate number is misleading because group A dominates the data and drowns out group B's errors. Next, check representation and label fairness in the training data, report per-group precision and recall as your real metric, and test remedies like resampling or class_weight="balanced" while watching the accuracy-versus-fairness tradeoff.

Q: You dropped the gender column, yet approval rates are still 80% for one group and 50% for another. What is happening, and how do you confirm it?

This is proxy discrimination in action: other features are standing in for gender, so the model keeps discriminating even though the column is gone. Confirm it by auditing the prediction distribution per group (the 80% versus 50% gap is disparate impact on its own), and by testing whether the remaining features can predict the sensitive attribute. If a quick classifier can recover gender from the "neutral" columns, you have proven the proxy exists and need mitigation beyond deletion.

Q: What does class_weight="balanced" do in a scikit-learn classifier, and what is the tradeoff?

It tells the model to weight each class inversely to how often it appears, so the minority class gets more say during training instead of being ignored in favor of the majority. This can lift recall for an underrepresented group and shrink an outcome gap. The tradeoff is that you usually give up a little overall accuracy and may raise false positives, so the right setting depends on the real-world cost of each error and who the model affects.

Series: Python + AI/ML Cookbook, Part 5: Machine Learning

Further reading: for the full reference, see the official Python documentation.

Related Posts

Previous: Debugging ML Models in Python: 5 Real Failure Cases

Next: ML: End-to-End Project, House Price Prediction

Series Home: Python + AI/ML Tutorial Series