Most interview rounds ask you to write a function; the ML system design round asks you to design a whole machine learning product out loud, and it is the stage that most often decides a senior offer. Given a vague goal like “recommend videos” or “catch fraud”, can you turn it into metrics, data, features, a model, a serving path, and monitoring? This post gives you a fixed skeleton, two fully worked designs, and a graded mock transcript.

“Anyone can train a model. The interview is asking whether you can keep one alive in production.”

What separates the design round from the coding round

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0, NumPy 2.4.6 | Difficulty: Expert | Reading Time: 21 minutes

Here is the everyday version. Say an architect named Aviraj is asked to design a house. He does not start by choosing paint colors, he starts with questions: how many people live here, what is the budget, which way does the sun face. Only then does he sketch rooms, plumbing, and wiring in a fixed order that every architect shares. The ML design round works the same way. The interviewer hands you a fuzzy goal, and they are watching whether you reach for a shared skeleton or start guessing at models. Learn the skeleton once and every question, recommender, fraud, search, ranking, ads, becomes the same seven rooms in a different house.

The picture above is the whole answer template, seven boxes you draw on the whiteboard before you say anything clever. Requirements come first, metrics second (both the offline number you optimize and the online number the business cares about), then data, features, model, serving, and monitoring, with a dotted feedback loop because a live model retrains as the world drifts. Everything below fills those boxes in, first as a checklist and then through two designs you can defend under follow-up questions.

The Whiteboard Skeleton

The single biggest mistake candidates make in ML system design is jumping straight to “I’d use a neural network.” A strong answer spends the first few minutes on requirements and metrics, because those decisions quietly control everything after them. Walk the seven boxes in order and narrate your reasoning. Requirements: who uses this, how many requests per second, and what is the latency budget, ten milliseconds for an ad auction is a different world from an overnight batch email.

Metrics: pick one offline metric you can compute on held-out data to iterate quickly, and name the online metric, measured by an A/B test on live traffic, that the business actually rewards. Those two rarely match, and saying so out loud is a green flag.

The remaining five boxes of the ML system design skeleton follow naturally. Data: where labels come from and whether they are honest. Features: how you compute inputs the same way in training and serving. Model: always a dull baseline first, then something richer only if it earns its keep. Serving: batch precompute versus real-time inference, chosen by the latency budget from box one. Monitoring: watch for drift and skew, and decide how often to retrain. Here is the offline half of the metrics box made concrete, ranking a handful of items and scoring the ranking the way a recommender is graded before it ever reaches live traffic.

📄 metrics_offline.py: precision@k and NDCG@k, the offline scores you iterate on

import numpy as np

# What the model scored for 8 items, and what the user actually clicked (1 = relevant)
scores    = np.array([0.9, 0.2, 0.8, 0.1, 0.75, 0.4, 0.6, 0.05])
relevant  = np.array([1,   0,   1,   0,   0,    1,   0,   0])

order = np.argsort(-scores)          # rank items by the model's score
ranked_rel = relevant[order]

def precision_at_k(rel, k):
    return rel[:k].sum() / k

def ndcg_at_k(rel, k):
    gains = rel[:k] / np.log2(np.arange(2, k + 2))
    ideal = np.sort(rel)[::-1][:k] / np.log2(np.arange(2, k + 2))
    return gains.sum() / ideal.sum()

for k in (3, 5):
    print(f"precision@{k}: {precision_at_k(ranked_rel, k):.2f}   ndcg@{k}: {ndcg_at_k(ranked_rel, k):.2f}")

print("offline: these scores rank held-out clicks. online: you still need an A/B test on live traffic.")

▶ Output

precision@3: 0.67   ndcg@3: 0.77
precision@5: 0.60   ndcg@5: 0.95
offline: these scores rank held-out clicks. online: you still need an A/B test on live traffic.

What happened here: Precision@3 of 0.67 says two of the top three ranked items were actually clicked, a blunt hit rate. NDCG cares about order too: it rewards putting relevant items near the top and discounts ones lower down by that log2 position weight, which is why NDCG@5 climbs to 0.95 once the ranking has surfaced most of the relevant items in a sensible order. These are the numbers you optimize offline because they are cheap to compute on logged clicks.

The honest caveat, and the sentence interviewers wait for, is that a higher NDCG does not guarantee more watch time or revenue, so the offline win only counts once an online A/B test confirms it. Naming both metrics, and admitting they can disagree, is the whole point of box two.

Worked Design 1: A Recommender in Two Stages

Picture a librarian helping a reader named Anvi find her next book. The library has a million titles, so the librarian does not read every spine. First she walks to the two or three shelves that match Anvi’s taste and grabs maybe fifty candidates, that is fast and rough. Then she reads the blurbs of just those fifty and hands over the best five, that is slow and careful.

Every large-scale recommender is built exactly this way, and the interview name for it is two-stage: a cheap retrieval step that narrows a huge catalog to a few hundred candidates, then an expensive ranking step that scores only those with a richer model. You cannot run a heavy model over a million items in fifty milliseconds, so you never try.

Here is that two-stage shape on a fifty-thousand-item catalog. Retrieval uses a cheap approximate index to pull five hundred candidates, then ranking scores only those. The question that matters is whether the cheap first stage threw away the items the user would have loved, so we measure recall against a full, exact scan.

📄 two_stage.py: retrieval narrows 50k to 500, ranking scores only those

import numpy as np

rng = np.random.default_rng(7)
n_items, dim = 50_000, 16          # a catalog of 50k items
item_vecs = rng.normal(size=(n_items, dim))
item_vecs /= np.linalg.norm(item_vecs, axis=1, keepdims=True)

# One user who liked 5 items; the "truth" is the top 20 a full, exact scan would pick
liked = rng.choice(n_items, 5, replace=False)
u = item_vecs[liked].mean(axis=0)

full_scores = item_vecs @ u
truth_top20 = set(np.argpartition(-full_scores, 20)[:20])

# STAGE 1 retrieval: a cheap approximate index. Does its 500-item pool keep the winners?
approx = full_scores + rng.normal(0, 0.02, n_items)   # index is not exact
candidates = np.argpartition(-approx, 500)[:500]
retrieval_recall = len(truth_top20 & set(candidates)) / 20

# STAGE 2 ranking: an expensive model runs on 500 candidates, never on all 50k
ranked = candidates[np.argsort(-(item_vecs[candidates] @ u))][:20]
final_recall = len(truth_top20 & set(ranked)) / 20

print(f"catalog size:               {n_items:,} items")
print(f"stage-2 scored only:        {len(candidates)} of {n_items:,} items ({len(candidates)/n_items:.1%})")
print(f"retrieval kept the winners: recall {retrieval_recall:.0%} of the true top 20")
print(f"final top-20 vs full scan:  recall {final_recall:.0%}")

# COLD START: a brand-new item has no learned vector, so personalised scoring is impossible
def serve(has_vector):
    return "personalised score" if has_vector else "popularity fallback (no embedding yet)"
print("brand-new item ->", serve(has_vector=False))

▶ Output

catalog size:               50,000 items
stage-2 scored only:        500 of 50,000 items (1.0%)
retrieval kept the winners: recall 100% of the true top 20
final top-20 vs full scan:  recall 100%
brand-new item -> popularity fallback (no embedding yet)

What happened here: The ranking model touched only 500 items, one percent of the catalog, yet retrieval still kept 100% of the true top twenty in its candidate pool, so the final list matches the full exact scan. That is the whole economic argument for two stages: near-full quality at a fraction of the compute, which is how a recommender answers in real time. The last line is the trap the interviewer will absolutely probe, cold start: a brand-new item or user has no learned vector, so you cannot score it personally, and you fall back to popularity or content features until it gathers history.

Two more follow-ups always come. Position bias: users click the top result partly because it is on top, not because it is best, so training on raw clicks teaches the model to favor whatever you already showed, and you correct it by logging the position and down-weighting or modeling it. Feedback loops: the model shapes what users see, which shapes the next batch of training data, so without fresh exploration the system narrows into a rut. For the full build, the recommender system deep dive walks the modeling end to end.

Worked Design 2: Fraud Detection

Fraud detection flips the recommender on its head, and interviewers love pairing them because the contrasts teach the framework. Think of airport security screening bags. Almost every bag is harmless, so a lazy scanner that waves everything through is right 99.9% of the time and catches nothing, that is the class-imbalance problem. Every bag must clear in seconds, that is the latency budget. And the security team, not the machine, decides how sensitive the alarm should be, because a missed weapon and a needless search cost wildly different amounts, that is the threshold as a business decision. Those three ideas, imbalance, latency, and a cost-driven threshold, are the spine of any fraud answer.

The imbalance and metric traps were drilled in the ML interview checkpoint, so the fresh idea worth showing here is the last one: the model outputs a probability, but choosing the cutoff that turns that probability into “block or allow” is an economics problem, not a modeling one. We attach a real cost to each mistake and let the money pick the threshold.

📄 fraud_threshold.py: the same model, a threshold chosen by dollars not by 0.5

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# A fraud-like problem: 1.5% of transactions are fraud
X, y = make_classification(n_samples=20_000, weights=[0.985, 0.015],
                           n_informative=6, random_state=1)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=1)
proba = LogisticRegression(max_iter=2000, class_weight="balanced").fit(Xtr, ytr).predict_proba(Xte)[:, 1]

# The business numbers, not the model, decide the threshold.
COST_FN = 250.0     # a missed fraud: the average chargeback we eat
COST_FP = 8.0       # a false alarm: a human reviews a good transaction

def cost_at(t):
    pred = proba >= t
    fp = int(np.sum(pred & (yte == 0)))
    fn = int(np.sum(~pred & (yte == 1)))
    return fp * COST_FP + fn * COST_FN, fp, fn

# Find the cost-optimal threshold on a fine grid
grid = np.round(np.arange(0.05, 0.96, 0.01), 2)
best_t = min(grid, key=lambda t: cost_at(t)[0])

print(f"{'thresh':>7} {'false+':>7} {'missed':>7} {'$ cost':>10}")
for t in (0.10, 0.30, 0.50, best_t, 0.90):
    c, fp, fn = cost_at(t)
    mark = "  <- cost-optimal" if t == best_t else ""
    print(f"{t:>7.2f} {fp:>7} {fn:>7} {c:>10,.0f}{mark}")

d_cost = cost_at(0.50)[0]
b_cost = cost_at(best_t)[0]
print(f"\nmoving the threshold from the default 0.50 to {best_t:.2f} saves ${d_cost-b_cost:,.0f}")
print("same model, same probabilities. only the business cutoff changed.")

▶ Output

 thresh  false+  missed     $ cost
   0.10    4227       9     36,066
   0.30    2050      19     21,150
   0.50     951      28     14,608
   0.76     221      39     11,518  <- cost-optimal
   0.90      33      73     18,514

moving the threshold from the default 0.50 to 0.76 saves $3,090
same model, same probabilities. only the business cutoff changed.

What happened here: At the naive 0.50 cutoff the system misses 28 frauds and raises 951 false alarms for a total cost of $14,608. Sweeping the threshold and pricing each error, a missed fraud at $250 against a review at $8, lands the optimum at 0.76, which tolerates a few more missed frauds to kill three quarters of the false alarms and saves $3,090 on this test slice alone. Nothing about the model changed, only the number where probability becomes action.

That is the sentence to say in the room: “the model gives me a probability, the business gives me the costs, and the threshold is where those two meet.” On the serving side you would add the latency budget, a fraud call blocks a live payment so it must return in tens of milliseconds, which pushes you toward a fast model and precomputed features rather than a heavy ensemble.

Feature Stores and Training/Serving Skew

This is the vocabulary that makes you sound like you have been on call for a model, not just built one in a notebook. Say a bank computes a feature “average spend over the last seven days.” The training team writes it in a batch pandas job. Months later the serving team, under deadline, re-implements the same feature in the live Application Programming Interface (API), and the two versions disagree in some small way: a different default for missing values, a different time window, a rounding difference.

The model was trained on one definition and is served another. That gap is training/serving skew, and it silently erodes accuracy in production while every offline test still looks fine, because the offline test uses the training definition. Here is that skew made painfully visible.

📄 skew.py: two code paths for one feature, and the silent AUC drop it causes

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

rng = np.random.default_rng(0)
n = 8000
avg_spend = rng.normal(60, 20, n)                    # a real signal
y = (rng.random(n) < 1 / (1 + np.exp(-(avg_spend - 60) / 12))).astype(int)

# 20% of rows arrive with the feature missing (a common reality)
missing = rng.random(n) < 0.20
train_mean = avg_spend[~missing].mean()              # the stat learned at training time

# TRAINING pipeline: impute the gaps with the training mean, the correct definition
X_train = np.where(missing, train_mean, avg_spend).reshape(-1, 1)
model = LogisticRegression().fit(X_train, y)
offline_auc = roc_auc_score(y, model.predict_proba(X_train)[:, 1])

# SERVING pipeline: a different team re-implements the feature and imputes missing with 0
X_serve = np.where(missing, 0.0, avg_spend).reshape(-1, 1)
live_auc = roc_auc_score(y, model.predict_proba(X_serve)[:, 1])

skewed_rows = float(np.mean(X_train.ravel() != X_serve.ravel()))
print(f"training-time impute value: {train_mean:.1f}   serving-time impute value: 0.0")
print(f"rows whose feature differs between the two paths: {skewed_rows:.0%}")
print(f"offline AUC (training path):  {offline_auc:.3f}")
print(f"live AUC    (serving path):   {live_auc:.3f}")
print(f"silent drop from training/serving skew: {offline_auc - live_auc:.3f}")

▶ Output

training-time impute value: 60.0   serving-time impute value: 0.0
rows whose feature differs between the two paths: 20%
offline AUC (training path):  0.799
live AUC    (serving path):   0.718
silent drop from training/serving skew: 0.081

What happened here: One in five rows got a different feature value in serving because a second code path filled missing values with 0 instead of the training mean of 60. Offline everything looked healthy at 0.799 AUC, but live performance quietly fell to 0.718, a 0.081 drop nobody would catch from the training metrics alone. This is why feature stores exist. A feature store, Feast is a widely used open-source one at the time of writing, is a single place that defines each feature once and serves the exact same computation to both training and the live model, so the two paths cannot drift apart.

Two more terms complete the vocabulary. A model registry versions your trained models with their metrics and lineage, so you can roll back a bad deploy and know exactly which data trained the model in production. And retrain cadence is your written answer to “how often do we refresh the model,” driven by how fast the data drifts, which connects straight to the model drift and monitoring lesson.

A Mock Interview, Graded

Reading an ML system design framework is not the same as hearing it under follow-up pressure. Below is a compressed transcript of the recommender question, the kind of back-and-forth a real round produces, followed by the rubric an interviewer scores you against. Notice that the interviewer keeps pushing on production traps, not on model architecture, because that is where senior candidates separate from junior ones.

Interviewer: Design a system to recommend videos on our home page. Where do you start?

Candidate: Before any model, requirements. How many videos in the catalog, how many requests per second, and what latency can the home page tolerate? I’ll assume millions of videos, high traffic, and a budget around 100 milliseconds. That budget alone tells me I cannot score the whole catalog per request, so I’ll go two-stage: cheap retrieval then heavier ranking.

Interviewer: Good. What do you optimize, and how do you know it works?

Candidate: Offline I optimize a ranking metric like NDCG on logged clicks so I can iterate fast. But the metric that decides a launch is online, long-term watch time measured by an A/B test, and those can disagree, so a model that wins offline still has to win the experiment before it ships.

Interviewer: A new creator uploads a video this morning. How does it ever get shown?

Candidate: Cold start. It has no interaction history, so its learned embedding is meaningless. I’d fall back to content features, topic, creator, thumbnail, and give it a small exploration budget so it earns real feedback, then switch to the personalized path once it has enough signal.

Interviewer: Your click model keeps recommending what it already shows. Why?

Candidate: Two coupled problems. Position bias, users click the top slot partly because it is on top, so I log position and account for it in training. And a feedback loop, the model shapes tomorrow’s training data, so I keep some exploration to avoid collapsing into a narrow rut.

Interviewer: Last one. Offline metrics look great for weeks, then engagement slips. What broke?

Candidate: My first suspect is training/serving skew or data drift, not the model math. I’d check whether a feature is computed differently in serving than in training, which a feature store would prevent, and whether inputs have drifted from the training distribution. Monitoring and a sensible retrain cadence catch both.

That transcript is not about knowing exotic models, it is about reaching for the skeleton and naming the trap before the interviewer does. Here is the rubric a panel typically uses, and it is the checklist to grade your own practice answers against.

Common Mistakes

❌ Mistake: Jumping to the model before the requirements

# Weak opening:
#   "I'd use a deep neural network with attention."   (on what data? what latency? what metric?)

# Strong opening:
#   1. Requirements: catalog size, requests/sec, latency budget
#   2. Metrics: offline proxy to iterate + online A/B metric that decides launch
#   ...only THEN pick a model, starting from a dull baseline

Why: The model is box five of seven, and choosing it early means every earlier decision, which the model depends on, gets made by accident. Naming an architecture in your first sentence signals you have trained models but not designed systems. Spend the opening minutes on requirements and metrics, and the model choice will often make itself.

❌ Mistake: Treating the classification threshold as fixed at 0.5

# Wrong: pred = proba >= 0.5          # 0.5 is a default, not a decision

# Right: pick the cutoff from the cost of each error
#   threshold = argmin over t of (false_positives * cost_FP + false_negatives * cost_FN)
#   a missed fraud and a false alarm rarely cost the same, so 0.5 is almost never optimal

Why: The 0.5 cutoff assumes a false positive and a false negative are equally costly, which is almost never true in a real product. As the fraud example showed, moving the threshold to match the real dollar costs saved thousands without touching the model. Stating that the threshold is a business decision, tuned on a validation set to the cost of each mistake, is a mark of someone who has shipped a classifier.

Best Practices

• Draw the seven boxes first. Requirements, metrics, data, features, model, serving, monitoring. Narrate them in order so the interviewer can follow your structure instead of guessing where you are going.
• Always start from a baseline. Popularity for a recommender, logistic regression for fraud. It sets a floor, ships fast, and forces any fancy model to prove it is worth the complexity.
• Say the offline metric and the online metric. One to iterate on, one that decides the launch, and admit out loud that they can disagree.
• Name the trap before the fix. Cold start, position bias, feedback loops, imbalance, training/serving skew, drift. Raising these unprompted is the clearest senior signal in the room.
• Quote tools as dated examples. The reasoning is evergreen, but specific tools move, so say a name like Feast “at the time of writing” and lean on the concept, one feature definition for training and serving, which will outlive any product.

Conclusion

ML system design stops being intimidating the moment you stop treating each question as new and start treating it as the same seven boxes in a different costume. You saw the skeleton, requirements to monitoring, then filled it twice: a two-stage recommender that scored one percent of a catalog yet kept every top result, and a fraud detector whose threshold was set by dollars rather than a default 0.5. You watched training/serving skew quietly drop a live model from 0.799 to 0.718 AUC, which is exactly the failure a feature store is built to prevent, and you read a graded transcript where the interviewer probed production traps, not architecture.

The tools will change, feature stores and registries will get new names, but the reasoning, and the discipline of naming the trap before the fix, is what the design round actually measures, so it is worth memorizing cold.

This post sits in the career and interviews chapter alongside the coding, ML, and Large Language Model (LLM) rounds. For the full path, from Python basics through the AI and machine learning deep dives, visit the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: Design a system to recommend products on an e-commerce home page. Where do you begin?

I begin with requirements, not a model: catalog size, requests per second, and the latency budget for the home page. A large catalog with a tight budget rules out scoring everything per request, so I go two-stage, cheap retrieval to a few hundred candidates then a richer ranking model over just those. I optimize an offline ranking metric like NDCG to iterate, but the launch decision comes from an online A/B test on a business metric such as revenue per session, and I flag cold start and feedback loops as the traps I’ll handle explicitly.

Q: A model that scored 94% offline performs terribly in production, and you notice features are computed by two different pipelines. Name the failure mode and the fix.

It is when the same feature is computed differently in the training pipeline and the live serving pipeline, a different missing-value default or time window, so the model is served inputs that do not match what it learned on. Offline metrics stay healthy while live accuracy silently falls. The fix is a feature store that defines each feature once and computes it the same way for both training and serving, backed by monitoring that compares live feature distributions against training.

Q: A fraud model has strong AUC. How do you turn it into a block-or-allow decision?

AUC ranks transactions but does not pick a cutoff, and the cutoff is a business decision. I attach a cost to each error, a missed fraud is the chargeback we absorb, a false alarm is a review plus a slightly annoyed customer, then choose the threshold that minimizes total cost on a validation set. Because those costs differ a lot, the best threshold is rarely 0.5, and I would revisit it as the fraud rate and costs change over time.

Q: Your offline metrics look great but live engagement dropped. Walk me through your debugging.

I suspect a production gap before I touch the model math. First I check training/serving skew, is any feature computed differently live than in training, which a feature store would catch. Then data drift, have the input distributions moved away from what the model trained on. Then the label and logging pipeline, are we still recording outcomes correctly. Monitoring on feature distributions and a sensible retrain cadence usually surface the real cause, which is far more often a pipeline problem than a modeling one.

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: How to Add AI to an Existing App Without Breaking It

Next: Retrieval-Augmented Generation (RAG) System Design: The LLM Interview Whiteboard Round

Series Home: Python + AI/ML Tutorial Series