Model drift is what happens when a machine learning model that scored 94% on launch day quietly slides to 70% six months later, without a single line of its code changing. The world moved, and the model stayed frozen in the past. This tutorial simulates that decay week by week, catches it early with Evidently, and wires up a monitoring loop that alerts and retrains before your users feel it.

“A model is a photograph of the world on the day you trained it. The world keeps moving. Your job is to notice when the photo stops matching.”

Last Updated: July 2026 | Tested on: Python 3.14.6, evidently 0.7.21, scikit-learn 1.9.0 | Difficulty: Advanced | Reading Time: 19 minutes

Most teams ship a model, celebrate the launch metrics, and then forget about it. That is exactly when model drift starts. A deployed model has no idea that mortgage rates jumped, that a competitor changed the market, or that your users started behaving differently. It keeps making confident predictions with stale logic. Catching that slide is the whole point of drift monitoring, and it is a favourite interview topic and audit checklist item for a reason. Everything below was run on Python 3.14.6 with Evidently 0.7.21 (the latest stable at the time of writing), so every number you see is the real output from the scripts, not a screenshot from someone else’s laptop.

Why Models Rot: Data, Concept, and Label Drift

Before we write code, get the model drift vocabulary straight, because interviewers love to check that you can tell these three apart. They all cause accuracy to fall, but for different reasons and with different fixes. Think of your model as a cook who learned every regular customer’s taste last year. Three separate things can make that memory go stale.

• Data drift (the inputs change): The distribution of the features shifts. Say a grocery delivery app trained its demand model when most orders were paneer and rice. A new season brings a wave of jackfruit and millet orders the model barely saw in training. The question the model answers is still valid, but it is now seeing inputs from a region it never learned well.
• Concept drift (the rules change): The relationship between the inputs and the answer changes. A spam filter learned that the word “free” signals junk mail. Then legitimate newsletters start using “free” in every subject line. Same word, opposite meaning now. The inputs may look normal while the correct answer for them has flipped.
• Label drift (the answer mix changes): The distribution of the target itself moves. A fraud model trained when 1% of transactions were fraud. During a festival sale, fraud jumps to 6%. Even if individual transactions look familiar, the base rate the model assumes is now wrong.

Here is the practical punchline. Data drift is the one you can catch immediately, because you can measure it the second a prediction comes in, no labels needed. Concept and label drift only show up once the true answers arrive, which is often days or weeks later. That timing gap is the whole reason monitoring architecture looks the way it does, and we will build exactly that below.

Simulate a Model Decaying Over 12 Weeks

Talking about drift is easy. Watching a model rot in front of you makes it stick. We will reuse the housing idea from the house price prediction project, but reframed as a classification task: will a house sell above the local median price? A data scientist named Aditi trains a Random Forest on a calm reference market where mortgage rates hover near 5% and buyers chase high-income neighbourhoods. Then we play the market forward twelve weeks. Rates climb from 5% to 9% (a real 2022-style shock), and as borrowing gets expensive, buyer behaviour flips: income stops mattering and the interest rate starts driving who buys what. The model never saw that world, so watch what happens.

📄 drift_simulation.py: watch accuracy decay week by week

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(42)

def features(n, rate_mean, rate_sd=0.8):
    sqft   = rng.normal(1800, 400, n)   # living area
    income = rng.normal(85, 20, n)      # local median income (thousands)
    rate   = rng.normal(rate_mean, rate_sd, n)  # mortgage rate (%)
    return sqft, income, rate

def demand_score(sqft, income, rate, w_income, w_rate):
    z_sqft   = (sqft - 1800) / 400
    z_income = (income - 85) / 20
    z_rate   = (rate - 5.0) / 1.5
    return 0.4 * z_sqft + w_income * z_income + w_rate * z_rate

X_cols = ["sqft", "income", "rate"]

# Reference (training) market: calm ~5% rates, buyers chase high-income areas.
sqft, income, rate = features(4000, rate_mean=5.0)
s = demand_score(sqft, income, rate, w_income=1.0, w_rate=-0.2) + rng.normal(0, 0.4, 4000)
ref = pd.DataFrame({"sqft": sqft, "income": income, "rate": rate,
                    "sold_above": (s > np.median(s)).astype(int)})

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(ref[X_cols], ref["sold_above"])
print(f"Reference market: mean rate {ref['rate'].mean():.2f}%, "
      f"positive rate {ref['sold_above'].mean():.1%}")
print()
print("Week | rate mean | income weight | positive rate | live accuracy")
print("-----|-----------|---------------|---------------|--------------")

for week in range(12):
    frac      = week / 11.0
    rate_mean = 5.0 + 4.0 * frac      # rates climb 5% -> 9% (data drift)
    w_income  = 1.0 - 1.2 * frac      # income stops mattering (concept drift)
    w_rate    = -0.2 - 1.2 * frac     # rate now drives buyer behaviour
    sqft, income, rate = features(1000, rate_mean=rate_mean)
    s = demand_score(sqft, income, rate, w_income, w_rate) + rng.normal(0, 0.4, 1000)
    y = (s > np.median(s)).astype(int)  # market stays ~balanced each week
    batch = pd.DataFrame({"sqft": sqft, "income": income, "rate": rate})
    acc = accuracy_score(y, model.predict(batch[X_cols]))
    print(f"  {week:>2} |   {rate_mean:5.2f}%  |     {w_income:+.2f}     "
          f"|    {y.mean():5.1%}     |    {acc:.3f}")

▶ Output

Reference market: mean rate 4.99%, positive rate 50.0%

Week | rate mean | income weight | positive rate | live accuracy
-----|-----------|---------------|---------------|--------------
   0 |    5.00%  |     +1.00     |    50.0%     |    0.874
   1 |    5.36%  |     +0.89     |    50.0%     |    0.856
   2 |    5.73%  |     +0.78     |    50.0%     |    0.858
   3 |    6.09%  |     +0.67     |    50.0%     |    0.832
   4 |    6.45%  |     +0.56     |    50.0%     |    0.798
   5 |    6.82%  |     +0.45     |    50.0%     |    0.753
   6 |    7.18%  |     +0.35     |    50.0%     |    0.702
   7 |    7.55%  |     +0.24     |    50.0%     |    0.621
   8 |    7.91%  |     +0.13     |    50.0%     |    0.610
   9 |    8.27%  |     +0.02     |    50.0%     |    0.536
  10 |    8.64%  |     -0.09     |    50.0%     |    0.531
  11 |    9.00%  |     -0.20     |    50.0%     |    0.467

What happened here: The model launched at 87.4% accuracy and slid to 46.7%, worse than a coin flip, over twelve weeks. Nobody touched the code. Notice the positive rate stayed pinned at 50% the whole time, so this is not the accuracy metric being fooled by an imbalanced class. Two forces combined: the interest rate distribution physically moved (data drift, the rate mean marching from 5% to 9%), and underneath it the buyer logic inverted (concept drift, the income weight sliding from +1.00 to negative).

This is the realistic case. A visible input shift usually travels together with a hidden change in what the inputs mean. The scary part is that if Aditi was only watching the app’s uptime and error logs, everything would look perfectly healthy while the predictions rotted.

Catch Drift with Evidently

Now the important question: could we have caught this model drift without waiting for accuracy to crater? Yes. Evidently is an open-source Python library that compares a reference dataset (your training data, the “known good” world) against a current batch (what production is seeing now) and tells you exactly which columns drifted and by how much. Think of it like a fitness tracker for your data: it does not fix anything, but it screams early when your numbers start trending the wrong way. Let us point it at week 0 versus week 11.

Evidently 0.7 uses a Dataset plus a DataDefinition to describe your columns, then runs a Report built from ready-made presets. Here Anvay generates two reports: a data-drift report on the raw features, and a classification-quality report that needs the true labels and the model’s predictions.

📄 evidently_reports.py: data drift and classification quality

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from evidently import Report, Dataset, DataDefinition, BinaryClassification
from evidently.presets import DataDriftPreset, ClassificationPreset

rng = np.random.default_rng(42)
X_cols = ["sqft", "income", "rate"]

def features(n, rate_mean, rate_sd=0.8):
    return rng.normal(1800, 400, n), rng.normal(85, 20, n), rng.normal(rate_mean, rate_sd, n)

def demand_score(sqft, income, rate, w_income, w_rate):
    return 0.4*(sqft-1800)/400 + w_income*(income-85)/20 + w_rate*(rate-5.0)/1.5

# Reference (week 0) and current (week 11), with real model predictions.
sqft, income, rate = features(4000, 5.0)
s = demand_score(sqft, income, rate, 1.0, -0.2) + rng.normal(0, 0.4, 4000)
ref = pd.DataFrame({"sqft": sqft, "income": income, "rate": rate,
                    "sold_above": (s > np.median(s)).astype(int)})
model = RandomForestClassifier(n_estimators=200, random_state=42).fit(ref[X_cols], ref["sold_above"])
ref["prediction"] = model.predict(ref[X_cols])

sqft, income, rate = features(1000, 9.0)
s = demand_score(sqft, income, rate, -0.2, -1.4) + rng.normal(0, 0.4, 1000)
cur = pd.DataFrame({"sqft": sqft, "income": income, "rate": rate,
                    "sold_above": (s > np.median(s)).astype(int)})
cur["prediction"] = model.predict(cur[X_cols])

# Report 1: data drift on the input features (no labels needed)
dd_feat = DataDefinition(numerical_columns=X_cols)
ref_f = Dataset.from_pandas(ref[X_cols], data_definition=dd_feat)
cur_f = Dataset.from_pandas(cur[X_cols], data_definition=dd_feat)
drift = Report([DataDriftPreset()]).run(cur_f, ref_f)

print("=== DATA DRIFT REPORT: week 11 vs reference ===")
for m in drift.dict()["metrics"]:
    name, val = m["metric_name"], m["value"]
    if isinstance(val, dict):
        print(f"columns drifted: {int(val['count'])}/3   share: {val['share']:.2f}")
    else:
        col  = name.split("column=")[1].split(",")[0]
        flag = "DRIFT" if val > 0.1 else "ok"
        print(f"  {col:<7} drift score {val:6.3f}   [{flag}]")

# Report 2: classification quality (needs target + prediction)
dd_cls = DataDefinition(
    numerical_columns=X_cols,
    classification=[BinaryClassification(target="sold_above", prediction_labels="prediction")],
)
ref_c = Dataset.from_pandas(ref[["sold_above", "prediction"]], data_definition=dd_cls)
cur_c = Dataset.from_pandas(cur[["sold_above", "prediction"]], data_definition=dd_cls)
perf = Report([ClassificationPreset()]).run(cur_c, ref_c)

print("\n=== CLASSIFICATION QUALITY: week 11 (live) ===")
wanted = ("Accuracy", "Precision(", "Recall(", "F1Score")
for m in perf.dict()["metrics"]:
    if m["metric_name"].startswith(wanted):
        print(f"  {m['metric_name'].split('(')[0]:<10} {m['value']:.3f}")

# Save the full interactive dashboards to open in a browser
drift.save_html("drift_report.html")
perf.save_html("performance_report.html")

▶ Output

=== DATA DRIFT REPORT: week 11 vs reference ===
columns drifted: 1/3   share: 0.33
  sqft    drift score  0.081   [ok]
  income  drift score  0.064   [ok]
  rate    drift score  4.986   [DRIFT]

=== CLASSIFICATION QUALITY: week 11 (live) ===
  Accuracy   0.487
  Precision  0.481
  Recall     0.332
  F1Score    0.393

What happened here: Read the drift report column by column. Evidently checked all three features and flagged one, rate, with a drift score of 4.986, miles past its 0.1 threshold (that number is a normalised Wasserstein distance, a measure of how far one distribution moved from another). The sqft and income columns barely budged, scores of 0.08 and 0.06, correctly marked "ok". So the very first signal, available the moment predictions arrive and needing no true labels at all, points straight at the interest rate.

The classification report then confirms the damage once labels are in: accuracy 0.487, and recall collapsed to 0.332, meaning the model now misses two out of every three houses that actually sold above median. The two reports tell a complete story. Data drift said "the rate feature moved", and the performance report said "and it wrecked our predictions". The save_html calls also write full interactive dashboards you can open in a browser and hand to a non-technical stakeholder.

A Monitoring Loop That Actually Fires

Running Evidently once in a notebook is a demo. Running it automatically, forever, on live traffic is monitoring. The architecture below is the standard shape every serious team converges on. The key insight is that timing gap we mentioned earlier: predictions are available instantly, but the ground-truth labels ("did the house actually sell above median?") show up days or weeks later, so the loop has to wait for them to join back up before it can judge real accuracy.

Follow the arrows. The live model logs every prediction along with its inputs and the exact model version. Separately, true labels trickle in later and get joined back onto those logged predictions. A scheduled job (this is where Evidently runs) fires daily or weekly and runs two checks: has the input data drifted, and has measured performance dropped? If everything is inside the thresholds, the loop simply keeps the current model serving.

If drift crosses the line, it alerts the on-call team, kicks off a retrain on fresh data, shadow-evaluates the new model against the old one on live traffic, and promotes the new model only if it genuinely wins. Then it loops back to serving. That last "only if it wins" gate is what stops you from replacing a tired model with a worse one.

Thresholds, Alert Fatigue, and Retrain Policies

The hardest part of model drift monitoring is not detecting drift. It is deciding when drift actually matters. Set the threshold too tight and you get a drift alert every Monday, everyone mutes the channel, and the one real emergency slips through. That is alert fatigue, and it kills more monitoring systems than bad code ever does. A useful rule of thumb: alert on sustained drift, not a single noisy batch. Requiring the drift score to stay high for, say, three runs in a row filters out the random weekly wobble and only fires on a real trend, like our rate marching from 5% to 9%.

Once an alert is real, you pick a retrain policy. There are three honest options, and each has its place.

Whichever you pick, never promote a retrained model on faith. Run it in shadow mode first: the new model makes predictions on the same live traffic as the old one, but only the old one's predictions are actually used. You compare them for a while, and you swap only if the new model measurably beats the old on the recent data. This is exactly where model versioning earns its keep, because promoting is just moving an alias, and rolling back is moving it right back. That is the MLflow registry workflow from the previous tutorial.

What to Log at Serve Time

None of this monitoring is possible unless your serving code logs the right things while it runs. This is the quiet decision that makes or breaks drift detection six months later, and it ties directly into the logging and FastAPI serving tutorials. For every prediction your Application Programming Interface (API) makes, write down four things.

• The input features exactly as the model saw them, after preprocessing. Without these there is nothing to compute data drift against.
• The output the model returned (the predicted class and, ideally, the probability), so you can later compare it to the truth.
• The model version that made the call, so when two versions overlap during a rollout you can tell which one produced which prediction.
• A prediction ID and timestamp, so when the real label arrives days later you can join it back to the exact prediction it belongs to.

📄 serve_logging.py: the minimum you log per prediction

import json, uuid
from datetime import datetime, timezone

def log_prediction(features: dict, pred: int, proba: float, model_version: str):
    record = {
        "prediction_id": str(uuid.uuid4()),
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "model_version": model_version,   # ties back to the MLflow registry
        "features": features,             # after preprocessing
        "prediction": pred,
        "probability": round(proba, 4),
        "label": None,                    # filled in later when truth arrives
    }
    print(json.dumps(record))             # in prod: write to a log store / table
    return record["prediction_id"]

pid = log_prediction(
    features={"sqft": 2100, "income": 78.5, "rate": 8.9},
    pred=0, proba=0.34, model_version="house-sale@v3",
)
print("logged prediction:", pid[:8])

▶ Output

{"prediction_id": "849c9e2c-e580-42ce-9f73-45a62ff67dde", "timestamp": "2026-07-09T19:23:44.990128+00:00", "model_version": "house-sale@v3", "features": {"sqft": 2100, "income": 78.5, "rate": 8.9}, "prediction": 0, "probability": 0.34, "label": null}
logged prediction: 849c9e2c

What happened here: Every prediction becomes one JSON line carrying its own ID, timestamp, and the model version that produced it. The label starts as null and gets filled in later, the moment you learn whether the house really sold above median. Line those records up against your reference training set and you can feed them straight into the Evidently reports from earlier. Skip this logging and you are flying blind: you will feel accuracy dropping but have no data to prove what changed or when.

Common Mistakes

Best Practices

• Freeze a reference dataset. Save the exact training distribution as your "known good" baseline and compare every production batch against it, so drift is always measured against the world the model actually learned.
• Log features, output, and model version on every prediction. This is the raw material for every report. Decide it on day one, because you cannot go back and log the past.
• Separate the two alarms. A fast data-drift alarm on the inputs, and a slower performance alarm once labels arrive. They answer different questions and you want both.
• Automate the report, keep a human on the trigger. Let the job run and post its dashboard automatically, but have a person approve the actual retrain-and-promote for anything high stakes.
• Version everything so rollback is one command. When a retrain goes wrong, moving an alias back to the previous model should take seconds, not a redeploy.

Conclusion

You watched a model drift from 87% down to a coin flip without one line of code changing, caught the cause with an Evidently data-drift report before the labels even arrived, confirmed the damage with a classification-quality report, and wired the whole thing into a monitoring loop that alerts, retrains, shadow-tests, and promotes only a winner. That is the full lifecycle of keeping a model honest in production, and it is what separates a model that quietly rots from one a team can trust for years.

Evidently is the tool we used here because it is open source and easy to run locally at the time of writing, but the taxonomy of data, concept, and label drift is evergreen. If your team standardises on NannyML, WhyLabs, or Arize instead, every model drift idea in this post carries straight over. Want the full picture? Browse the complete Python + AI/ML tutorial series home to see how drift monitoring fits alongside the training, deployment, and MLflow tutorials.

More in this series:

• Python ML Model Serving: From Notebook to Production API
• ML: Best Practices, the Ethics and Catches Nobody Warns You About
• ML: End-to-End Project, House Price Prediction

Frequently Asked Questions

Interview Questions on Model Drift

Try each one aloud first. The phrasing you produce under mild pressure is what interviews measure.

Q: Define data drift, concept drift, and label drift with an example of each.

Data drift is a shift in the input distribution, for example a demand model suddenly seeing a season of jackfruit orders it rarely trained on. Concept drift is a change in the input-to-target relationship, like a spam filter after the word "free" starts appearing in legitimate newsletters. Label drift is a shift in the target distribution, such as fraud jumping from 1% to 6% during a festival sale. All three lower accuracy, but data drift is detectable from inputs alone while concept and label drift need the true outcomes.

Q: Your model's accuracy dropped in production but the input features look unchanged. What kind of drift is this?

That points to concept drift. The inputs still look like the training data, so a data-drift check would come back clean, but the correct answer for those same inputs has changed. You would catch this only through a performance drop once true labels arrive, which is exactly why relying on input-drift monitoring alone is not enough.

Q: Why can you detect data drift immediately but not concept drift?

Data drift is measured purely on the input features, which you have the instant a prediction is made, so you compare the live batch against the reference distribution right away. Concept drift is about whether predictions are still correct, and correctness needs ground-truth labels, which usually arrive days or weeks later. That label delay is the fundamental reason the two alarms run on different clocks.

Q: How do you avoid alert fatigue in a drift monitoring system?

Do not fire on a single noisy batch. Require the drift to be sustained across several consecutive runs before alerting, so random weekly variation is filtered out and only a real trend triggers a page. Also separate an informational dashboard from an actual page, and tie the alert to a business-relevant metric drop rather than any statistical wobble, so the humans keep trusting the alarm.

Q: What is shadow evaluation and why use it before promoting a retrained model?

Shadow evaluation runs the new model on the same live traffic as the current one, but its predictions are not used to make decisions yet. You collect both sets of predictions, wait for the true labels, and compare. You promote the new model only if it measurably beats the incumbent on recent data. This prevents the common failure of replacing a tired model with one that is actually worse, and because promotion is just moving a version alias, rollback is instant if something slips through.

Q: What must you log at serve time to make drift monitoring possible later?

At minimum: the input features after preprocessing, the model's output (predicted class and probability), the model version that produced it, and a prediction ID with a timestamp. The features feed data-drift checks, the model version disambiguates overlapping rollouts, and the ID plus timestamp let you join the true label back to the exact prediction once it arrives so you can measure real accuracy.

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: Python: Machine Learning Operations (MLOps) with MLflow, Experiment Tracking and Model Versioning

Next: Debugging ML Models in Python: 5 Real Failure Cases

Series Home: Python + AI/ML Tutorial Series