RAG system design, short for Retrieval-Augmented Generation, is the GenAI whiteboard round where an interviewer says “design ChatPDF for an enterprise” and then watches how you think for forty-five minutes, not what you can draw. This post walks that round end to end: an eight-layer framework, two fully worked designs, the follow-up gauntlet on ACLs, freshness, and scale, and a mock transcript with the rubric, every number computed with real code.

“They grade the numbers you defend, not the boxes you draw.”

What the GenAI system-design round is really testing

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0, NumPy 2.4.6 | Difficulty: Expert | Reading Time: 23 minutes

Here is the everyday version. Say a senior engineer named Anvay is asked to plan a new library for a school. A weak plan is a pretty floor map with shelves everywhere. A strong plan answers the boring questions: how many books arrive each week, who is allowed into the reference room, how fast can a student find one title, and what does the whole thing cost to run each month.

The system-design round is the same. The panel is not impressed by a neat diagram of arrows between boxes, they are checking whether you reason about cost, latency, evaluation, and monitoring, because those are what decide if a retrieval system survives contact with real users. Draw the boxes fast, then spend your time on the numbers under them.

The picture above is the whole answer in one frame: a corpus flows down through ingestion, retrieval with access control, and grounded generation to an answer, while evals, observability, and a cost model wrap the entire path and feed the grade. Notice what sits at the bottom, config-pinned component ids, because the model you name today will be deprecated before your offer letter dries, and a good design survives that. Everything below walks this map one layer at a time.

How the GenAI System-Design Round Works

The RAG system design prompt is deliberately open. “Design a system that lets employees ask questions over ten thousand internal PDFs” has no single right answer, and that is the point. The panel wants to hear you turn a vague ask into concrete requirements, then make tradeoffs out loud. From reading how these rounds are scored across companies, the pattern is blunt: candidates who spend the hour drawing a prettier architecture lose to candidates who draw an ugly one in five minutes and then talk about cost, latency, evaluation, and monitoring for the rest. Box-drawing is table stakes.

The signal is whether you can say “this design costs about eight thousand dollars a month, holds a three-second p95, and blocks a deploy when retrieval recall drops,” and back each figure.

So run the round in two moves. First, spend three minutes pinning requirements: corpus size, number of tenants, queries per day, freshness needs, and the latency and cost budgets. Second, walk the eight layers below, pausing at each to name the decision and its tradeoff. Keep the specific product names, embedding models, and vector stores loose, because those change every quarter, and anchor on the durable decisions, which do not.

The Eight-Layer Design Framework

The diagram above is eight layers you can recite in order: corpus and scale, ingestion, retrieval, generation, evals, observability, cost, and guardrails. The first four are the request path a query travels, and the last four wrap around it so the thing stays correct and affordable. Most candidates can describe the request path. What separates a strong RAG system design answer is treating the wrapping layers as first-class, because that is exactly where the interviewer’s questions are going.

Start with the two numbers that decide everything downstream: what does a query cost, and how long does it take. Put both on the board as a budget broken out by stage, because a single blended number hides where the money and the milliseconds actually go. Here is that budget for a mid-sized deployment, computed rather than guessed.

📄 rag_budget.py: the cost and latency budget an interviewer wants on the board

# RAG whiteboard: the cost and latency budget, stage by stage, per query.
# Prices are illustrative "at the time of writing" numbers, the framework is
# what you defend, not the digits.
queries_per_day = 100_000
avg_context_tokens = 2_400   # retrieved chunks stuffed into the prompt
avg_output_tokens = 350      # the model's answer

embed_price = 0.02           # per 1M tokens (USD), illustrative
gen_input_price = 0.60
gen_output_price = 2.40
rerank_price_per_query = 0.0005

embed_tokens = 60            # the user question we embed to search
def usd(tokens, price_per_m):
    return tokens / 1_000_000 * price_per_m

c_embed  = usd(embed_tokens, embed_price)
c_rerank = rerank_price_per_query
c_gen_in = usd(avg_context_tokens, gen_input_price)
c_gen_out = usd(avg_output_tokens, gen_output_price)
per_query = c_embed + c_rerank + c_gen_in + c_gen_out

print("cost per query (USD):")
print(f"  embed query      {c_embed:.6f}")
print(f"  rerank           {c_rerank:.6f}")
print(f"  generate (input) {c_gen_in:.6f}   <- the retrieved context dominates")
print(f"  generate (output){c_gen_out:.6f}")
print(f"  total/query      {per_query:.6f}")
print(f"daily cost:  ${per_query * queries_per_day:,.0f}")
print(f"monthly:     ${per_query * queries_per_day * 30:,.0f}")

stages = {"embed": 40, "vector search": 90, "rerank": 220, "generation": 1900}
p95 = sum(stages.values())
slo = 3000
print("\nlatency budget (p95, ms):")
for name, ms in stages.items():
    print(f"  {name:<14} {ms:>5}")
print(f"  {'sum':<14} {p95:>5}  vs SLO {slo}  -> {'PASS' if p95 <= slo else 'FAIL'}")
print(f"  headroom left: {slo - p95} ms (retries, network, cold cache eat this)")

▶ Output

cost per query (USD):
  embed query      0.000001
  rerank           0.000500
  generate (input) 0.001440   <- the retrieved context dominates
  generate (output)0.000840
  total/query      0.002781
daily cost:  $278
monthly:     $8,344

latency budget (p95, ms):
  embed             40
  vector search     90
  rerank           220
  generation      1900
  sum             2250  vs SLO 3000  -> PASS
  headroom left: 750 ms (retries, network, cold cache eat this)

What happened here: Two facts on the board reframe the whole design. First, the retrieved context is the dominant cost, not the answer the model writes, because you pay for every token you stuff into the prompt and a fat top-k of chunks adds up fast. That single line tells the interviewer why you would rerank down to fewer, better chunks rather than pad the prompt: it saves money and latency at once.

Second, generation eats the latency budget, 1900 of the 2250 milliseconds, leaving only 750 milliseconds of headroom under a three-second SLO for retries, network hops, and a cold cache. So if someone later asks “where would you add streaming or a smaller model,” you already know the answer lives in that generation stage. Naming these two pressure points early makes every later tradeoff sound deliberate instead of reactive.

With cost and latency framed, the other layers slot in. Ingestion decides chunk size and how you tag each chunk with its owner and timestamp. Retrieval is where hybrid search (dense embeddings plus keyword) and a reranker earn their keep, covered fully in the agentic RAG post. Generation grounds the answer in retrieved text and cites it. Then the wrapping layers: evals gate deploys on a golden set, observability traces cost and latency per query as in the Large Language Model (LLM) observability post, the cost model above keeps the bill honest per the cost optimization post, and guardrails from the AI guardrails post refuse ungrounded or unsafe answers.

Worked Design One: ChatPDF for an Enterprise

Now the real prompt: build ChatPDF for a company where every team uploads its own documents and nobody should see another team’s files. The moment you make it multi-tenant, the most important line in the whole system is not the ranker, it is the access filter, and where you put it decides whether you ship a data leak. The rule is simple and interviewers listen for it: filter by who is allowed to see a chunk before you rank, never after. Filtering after ranking means the model already saw forbidden text, and a reranker that “usually” drops it is not a security control. Here is the filter running alongside a freshness check, because stale chunks are their own quiet failure.

📄 acl_freshness.py: filter by tenant and ACL before ranking, and flag stale chunks

# The follow-up gauntlet made concrete: multi-tenant ACLs + freshness.
# Rule 1: filter by who may see a chunk BEFORE ranking, never after.
# Rule 2: a chunk older than its source's re-index cadence is stale.
from datetime import date
today = date(2026, 7, 10)

chunks = [
    {"id": 0, "tenant": "acme",  "acl": "all",     "indexed": date(2026, 7, 8), "text": "Acme refund window is 30 days."},
    {"id": 1, "tenant": "acme",  "acl": "finance", "indexed": date(2026, 6, 1), "text": "Acme Q2 revenue was 4.2 cr."},
    {"id": 2, "tenant": "globex","acl": "all",     "indexed": date(2026, 7, 9), "text": "Globex refund window is 15 days."},
    {"id": 3, "tenant": "acme",  "acl": "all",     "indexed": date(2026, 3, 2), "text": "Acme old refund window is 45 days."},
]
cadence_days = 30   # how fresh a chunk must be to be trusted

def visible(user, chunks):
    return [c for c in chunks
            if c["tenant"] == user["tenant"]
            and (c["acl"] == "all" or c["acl"] in user["groups"])]

def freshness(chunks):
    for c in chunks:
        c["age"] = (today - c["indexed"]).days
        c["stale"] = c["age"] > cadence_days
    return chunks

aditi = {"name": "Aditi", "tenant": "acme", "groups": {"support"}}
allowed = freshness(visible(aditi, chunks))

print(f"user Aditi (tenant=acme, groups={aditi['groups']}) can retrieve:")
for c in allowed:
    tag = "STALE, drop or re-index" if c["stale"] else "fresh"
    print(f"  chunk {c['id']}  age {c['age']:>3}d  [{tag}]  {c['text']}")

blocked = [c["id"] for c in chunks if c not in allowed]
print(f"\nblocked before ranking: chunks {blocked} (other tenant + finance-only)")
fresh_ids = [c["id"] for c in allowed if not c["stale"]]
print(f"served to the model: chunks {fresh_ids} (visible AND fresh)")

▶ Output

user Aditi (tenant=acme, groups={'support'}) can retrieve:
  chunk 0  age   2d  [fresh]  Acme refund window is 30 days.
  chunk 3  age 130d  [STALE, drop or re-index]  Acme old refund window is 45 days.

blocked before ranking: chunks [1, 2] (other tenant + finance-only)
served to the model: chunks [0] (visible AND fresh)

What happened here: Aditi belongs to the acme tenant and the support group, so two chunks vanish before anything is scored: chunk 2 is another company’s data, and chunk 1 is finance-only. They are never embedded into a query, never ranked, never near the model, which is what “secure by construction” means rather than “secure by ranking.” Of the two chunks she can see, one is two days old and one is one hundred and thirty days old, well past the thirty-day cadence, so it is flagged stale.

The old refund policy saying forty-five days is exactly the kind of chunk that produces a confidently wrong answer, so freshness drops it and leaves only chunk 0. When the interviewer asks “how do you handle a policy that changed last week,” you point at this: re-index on a cadence, stamp each chunk, and let anything past its cadence be dropped or refreshed before it can mislead a user.

The Follow-Up Gauntlet

Once your design stands, the interviewer starts pushing. This is the gauntlet, a rapid series of “what if” questions, and the trap is answering each with a vague “we would monitor that.” The panel wants a mechanism. Multi-tenant ACLs you have already shown. Freshness you have shown. That leaves the hallucination budget, ten-times scale, and the latency SLO, and the strongest single answer to most of them is an evaluation gate: no configuration change reaches production unless retrieval quality on a fixed golden set clears a bar. Here is that gate catching a bad chunking change before it ships.

📄 eval_gate.py: block a deploy when retrieval recall drops, plus a hallucination budget

# The eval + guardrail layer as a CI gate: no config ships unless retrieval
# recall@k on a golden set clears a threshold.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

corpus = [
    "Paid leave carries over up to 6 days into the following year.",     # 0
    "Sick leave is a separate pool and does not carry over at all.",     # 1
    "Reimbursement claims must be filed within 30 days of purchase.",    # 2
    "The notice period for senior engineers is 60 days.",                # 3
    "Health insurance covers the employee, a spouse, and two children.", # 4
    "Work-from-home is allowed up to three days a week with approval.",  # 5
]
golden = [
    ("how many leave days carry into next year?", 0),
    ("does sick leave carry over?", 1),
    ("what is the reimbursement filing deadline?", 2),
    ("how long is a senior engineer's notice period?", 3),
    ("who does health insurance cover?", 4),
]

def recall_at_k(chunk_size_ok, k=3):
    docs = list(corpus)
    if not chunk_size_ok:
        # a bad config over-splits the reimbursement rule: the deadline orphan
        # loses the topic word, so the query can no longer find it
        docs[2] = "Company expense and claims policy, general overview section."
        docs.append("This must happen within 30 days of the purchase date.")
    vec = TfidfVectorizer(stop_words="english")
    mat = vec.fit_transform(docs)
    hits = 0
    for q, goldid in golden:
        top = cosine_similarity(vec.transform([q]), mat)[0].argsort()[::-1][:k]
        hits += goldid in top
    return hits / len(golden)

THRESHOLD = 0.90
for label, ok in [("current config", True), ("proposed change", False)]:
    score = recall_at_k(ok)
    verdict = "SHIP" if score >= THRESHOLD else "BLOCK deploy"
    print(f"{label:<16} recall@3 = {score:.0%}   gate({THRESHOLD:.0%}) -> {verdict}")

queries, groundedness, budget = 10_000, 0.985, 100
ungrounded = round(queries * (1 - groundedness))
print(f"\nhallucination budget: {ungrounded} ungrounded / {queries} queries "
      f"(budget {budget}) -> {'within budget' if ungrounded <= budget else 'OVER budget'}")

▶ Output

current config   recall@3 = 100%   gate(90%) -> SHIP
proposed change  recall@3 = 80%   gate(90%) -> BLOCK deploy

hallucination budget: 150 ungrounded / 10000 queries (budget 100) -> OVER budget

What happened here: Someone proposes a new chunking config that over-splits the reimbursement rule, separating the “within 30 days” deadline from the word that names it. On the golden set, recall@3 quietly slips from 100% to 80%, meaning one in five questions no longer retrieves its answer, and the gate blocks the deploy before a single user sees a degraded answer. That is the difference between a demo and a system: the demo would have shipped and the regression would surface as angry tickets a week later.

The hallucination budget line makes the same idea a number you negotiate up front: at 98.5% groundedness you produce 150 ungrounded answers per 10,000 queries, which is over a budget of 100, so either you raise groundedness with stricter retrieval and refusal, or you accept and monitor the gap. For the ten-times-scale question, you point back at the cost model and say what shards, replicas, and caching you add per stage; the gate and the budget are what keep quality flat while the traffic grows.

Worked Design Two: An Internal Support Agent

The second design the panel likes to pivot to is an internal support agent: same retrieval spine, but now the system can also take actions, like looking up an order status or opening a ticket. Two layers get added on top of the RAG design. First, tools, and the durable way to expose them is a shared protocol so the same tool works across models rather than being wired to one vendor, which is the Model Context Protocol covered earlier in the series.

You describe each tool once, and the model decides when to call it. Second, and this is the answer interviewers wait for, a human in the loop for anything that writes or spends. The retrieval and grounding you already designed handle the reading; the moment an action has side effects, it routes to a person for approval, which is the pattern from the AI agent project post.

So the design reads: retrieve grounding context exactly as before, let the agent propose an action as a structured tool call, and gate any state-changing call behind a human approval step with the proposed action shown in plain language. Read-only tools like an order lookup can run automatically; a refund or a config change waits for a click. Say that split out loud and you have shown the panel you know agents are useful and dangerous in the same breath, which is exactly the maturity the round is scoring.

Designing for the Model of the Month

A quiet part of the RAG system design rubric is whether your design rots the day a model is deprecated. The interviewer may not ask it directly, but if you hard-code a specific model name into every layer, a sharp panel notices. The fix is small and worth saying out loud: pin every swappable component behind a config id, so upgrading a model is a one-line change plus a re-run of the eval gate, not a rewrite. Decisions are evergreen, component names are configuration.

📄 model_churn.py: pin components behind config so a model swap is one line

# Dead-tech-proofing: pin every swappable component behind a config id, so the
# day a model is deprecated you change one line, not the pipeline.
PIPELINE = {
    "embed_model":  "embed-v3-1024",      # any provider's embedder, pinned by id
    "reranker":     "cross-encoder-mini",
    "generator":    "gen-medium-2026-06", # date-stamped so a swap is auditable
    "vector_store": "hnsw-cosine",
}

def build_pipeline(cfg):
    return (f"ingest -> embed[{cfg['embed_model']}] -> store[{cfg['vector_store']}] "
            f"-> retrieve -> rerank[{cfg['reranker']}] -> generate[{cfg['generator']}]")

print("today's pinned pipeline:")
print(" ", build_pipeline(PIPELINE))

new_cfg = {**PIPELINE, "generator": "gen-medium-2026-09"}  # a newer generator ships
print("\nafter swapping the generator (one line):")
print(" ", build_pipeline(new_cfg))

changed = [k for k in PIPELINE if PIPELINE[k] != new_cfg[k]]
print(f"\ncomponents touched by the swap: {changed}")
print("retrieval, chunking, ACLs, evals: all unchanged and still valid")

▶ Output

today's pinned pipeline:
  ingest -> embed[embed-v3-1024] -> store[hnsw-cosine] -> retrieve -> rerank[cross-encoder-mini] -> generate[gen-medium-2026-06]

after swapping the generator (one line):
  ingest -> embed[embed-v3-1024] -> store[hnsw-cosine] -> retrieve -> rerank[cross-encoder-mini] -> generate[gen-medium-2026-09]

components touched by the swap: ['generator']
retrieval, chunking, ACLs, evals: all unchanged and still valid

What happened here: The whole pipeline is described by a config dictionary, and every stage reads its component from that config instead of a hard-coded name. When a newer generator ships, you change one string, the date-stamped id makes the swap auditable in review, and the diff shows exactly one component touched. Retrieval, chunking, ACLs, and evals are untouched and still valid, so you re-run the eval gate against the new generator and ship only if quality holds.

This is the concrete version of the durable advice this whole series repeats: at the time of writing the specific model names will be different from whatever is current when you read this, so anchor the design on the decisions and let the names live in config where they belong.

Mock Transcript and Rubric

Here is how the first four minutes of a RAG system design round should sound, compressed. It is not about having every answer, it is about the order you reach for things.

Interviewer: Design ChatPDF for our enterprise.
Anvay: Before I draw anything, four numbers: how many documents, how many separate teams sharing the system, roughly how many questions a day, and how fresh the answers need to be. Say ten thousand PDFs, forty teams, a hundred thousand questions a day, and policies that change weekly.
Interviewer: Good. Go.
Anvay: Request path is ingest, retrieve, generate. Ingestion chunks each PDF and tags every chunk with its team and an index date.

Retrieval does hybrid search, reranks to a handful of chunks, and critically filters by team before ranking so nobody sees another team’s files. Generation grounds the answer in those chunks and cites them. Around that: a golden-set eval gate on every change, tracing for cost and latency per query, and guardrails that refuse when nothing relevant is retrieved.
Interviewer: What does it cost?
Anvay: About eight thousand dollars a month at this volume, and the retrieved context is the dominant term, so I would rerank down rather than stuff the prompt.

The rubric behind that exchange is short. Did the candidate pin requirements before drawing? Did they put the access filter before ranking? Did they name real cost and latency numbers? Did they gate deploys on evaluation? Did they mention monitoring and guardrails without being prompted? Five yeses is a strong hire signal, and notice not one of them is about drawing a prettier diagram. The full build behind this round is the series capstone, the AI capstone project, which is the take-home proof that you can ship what you just whiteboarded.

Common Mistakes

❌ Mistake: Filtering access after ranking instead of before

# Wrong: rank all chunks, then drop the ones the user is not allowed to see.
#   The forbidden text was already embedded, scored, and sometimes cached.
#   "The reranker usually removes it" is not a security control.

# Right: filter by tenant and ACL group FIRST, so forbidden chunks are never
#   scored and never reach the model. Secure by construction, not by ranking.

Why: Access control that runs after retrieval is a leak waiting for an off-by-one. If a chunk was in the candidate set, it influenced scores, may sit in a cache, and is one bug away from appearing in an answer. Filtering first, by tenant and group, means forbidden data is structurally absent from the pipeline, which is the only story that survives a security review. Interviewers treat the order of these two steps as a direct signal of whether you have shipped a multi-tenant system before.

❌ Mistake: Spending the hour on the diagram instead of the numbers

# Wrong: keep refining boxes and arrows, add a caching box, a queue box,
#   a nicer layout, and never say what anything costs or how it is measured.

# Right: draw an ugly diagram in five minutes, then spend the rest on cost per
#   query, the latency budget, the eval gate, and what you monitor in production.

Why: The round is scored on reasoning about cost, latency, evaluation, and monitoring, and every minute spent beautifying the diagram is a minute not spent on the things that actually move the grade. A rough sketch plus “here is the cost model, here is the p95 budget, here is the deploy gate” reads as someone who runs these systems. A gorgeous diagram with no numbers reads as someone who has only read about them.

Best Practices

• Pin requirements before drawing. Corpus size, tenants, queries per day, freshness, and the cost and latency budgets. Three minutes here makes every later tradeoff concrete instead of hypothetical.
• Put the access filter before ranking. In a multi-tenant system this one ordering decision is the difference between a design and a data leak, and interviewers listen for it specifically.
• Name real numbers. Say the monthly cost, the p95 budget, and the recall threshold out loud. “About eight thousand a month, the context tokens dominate” beats “it depends” every time.
• Gate deploys on evals. A golden set plus a recall threshold that blocks a bad change is the single most convincing thing you can add, because it proves you keep quality flat as the system evolves.
• Keep component names in config. Pin embedder, reranker, and generator behind ids, and say “at the time of writing” for any specific model, so the design outlives the model of the month. Verified on Python 3.14.6 and scikit-learn 1.9.0 at the time of writing.

Wrapping Up

RAG system design stops being intimidating once you see it as an eight-layer framework with the numbers on top: pin requirements, sketch the request path of ingest, retrieve, and generate, then spend your real time on the wrapping layers of evals, observability, cost, and guardrails. You computed the pieces people fumble instead of reciting them: a query costing about a quarter of a cent where the retrieved context dominates the bill, a latency budget with only 750 milliseconds of headroom under a three-second SLO, an access filter that blocks forbidden chunks before they are ever scored, an eval gate that catches an 80% recall regression before it ships, and a config swap that upgrades a model in one line.

Answer the follow-up gauntlet with mechanisms, not with “we would monitor that,” and the round turns from an interrogation into a conversation you are leading. The durable caveat holds here too: the model names drift, so anchor on the decisions and let the names live in config.

This round is where the whole generative AI half of the series comes together, from retrieval and evals to observability and guardrails. For the full roadmap, from beginner basics through the AI/ML deep dives, visit the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: An interviewer says “design ChatPDF for our enterprise.” What are your first words?

Not a diagram. I pin requirements first: how many documents, how many separate teams share the system, roughly how many questions a day, and how fresh answers must be, plus the cost and latency budgets. Those numbers decide everything downstream, so I get them on the board before I draw a single box. Only then do I sketch the request path of ingest, retrieve, and generate, and I spend the real time on the wrapping layers of evals, observability, cost, and guardrails, because that is what the round actually grades.

Q: Where do you put access control in a multi-tenant RAG system, and why does the order matter?

Before ranking, always. I filter chunks by tenant and access group first, so a user’s query only ever scores documents they are allowed to see. If I filtered after retrieval instead, the forbidden text was already embedded, scored, and maybe cached, and a reranker that usually drops it is not a security control, it is a hope. Filtering first makes forbidden data structurally absent from the pipeline, which is the only version that survives a security review. That ordering is the whole answer, and interviewers listen for it specifically.

Q: How would you stop a bad change from silently degrading retrieval quality?

A golden set and a deploy gate. I keep a fixed set of questions each paired with the chunk that truly answers it, and I measure retrieval recall at k on every configuration change. If a new chunk size or embedder drops recall below the threshold, say ninety percent, the gate blocks the deploy before any user is affected. I showed this catching an over-splitting change that dropped recall from a hundred to eighty percent. Without that gate the regression ships and comes back as support tickets a week later, which is exactly the failure the gate exists to prevent.

Q: The system needs to handle ten times the traffic next quarter. Walk me through it.

I go back to the per-stage budget rather than guessing. Retrieval scales horizontally with more index shards and read replicas, and I add a cache for repeated queries since support traffic is heavy on duplicates. Generation is the expensive stage in both cost and latency, so I look at a smaller or distilled model for common questions, streaming to hide latency, and continuous batching on the serving layer. Throughout, the eval gate keeps quality flat and the cost model tells me whether the new footprint still pencils out. Ten-times traffic is a capacity and caching problem per stage, not a redesign.

Go deeper: when you outgrow this post, the official Python documentation is the next stop.

Related Posts

Previous: ML System Design: Recommenders, Feature Stores, Skew

Next: AI Project in Python: Building a Complete AI Application

Series Home: Python + AI/ML Tutorial Series