Anyone can wire a Large Language Model (LLM) to a folder of documents and get answers back. Proving those answers are good is a different job, and it is the one interviewers actually probe. This RAG project is the capstone for the GenAI chapter: build a document-question app, score it with real numbers, trace every request, and gate every merge behind CI.

“If you cannot measure it, you cannot improve it, and you certainly cannot ship it.”

Last Updated: July 2026 | Tested on: Python 3.14.6, llama-index 0.14.22, ragas 0.2.x, langfuse 3.x | Difficulty: Advanced | Reading Time: 24 minutes

Everything in this chapter has been a separate skill so far: Retrieval-Augmented Generation (RAG) with LlamaIndex, hybrid search and reranking, observability with Langfuse, and the CI eval gate pattern. This post wires them into one thing that works and, more importantly, one thing you can prove works. Think of it like a restaurant health inspection. The kitchen already cooks fine on a normal day. The inspector shows up, checks the fridge temperatures, tastes the food against a checklist, and either signs the certificate or shuts you down. Our checklist is a set of acceptance criteria, and we tick every box with code you can run.

The whole pipeline is provider-pluggable on purpose. Every model id and provider lives in a config file, never hardcoded, so when a model is renamed or a cheaper one appears you change one line, not the code. At the time of writing we use OpenAI embeddings, a Cohere reranker, and an OpenAI generator, but the stack table near the end lists a named swap for every single component. The blocks that need a paid Application Programming Interface (API) key or a running Langfuse server are marked as example output so you can read them honestly.

Everything else, the retrieval math, the golden-set validation, the trace parsing, and the CI gate, was really run on Python 3.14.6 and shows the real captured bytes.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

The left column is the answer pipeline that runs at request time: a question fans out to a keyword retriever and a dense (embedding) retriever, their two ranked lists are fused and reranked, and the top passages go to the model, which writes an answer with citations. Every step is traced by Langfuse. The right column is the quality gate that runs in CI: a 25-question golden set is scored by Ragas, the scores are written to a file committed in the repo, and a gate compares them to thresholds. Clear the floor and the merge ships; fall short and the pull request is blocked. Those two loops, the answer and the gate, are the whole RAG project.

What You Are Building (Acceptance Criteria)

Professionals agree on what “done” means before they start, so nobody argues about it later. It is like a caterer confirming the menu, the headcount, and the delivery time in writing before the event. Here is the checklist for this RAG project. By the end of this post, every item is ticked with code you ran yourself.

1. Answers a question over a user-supplied folder of documents, with hybrid retrieval (keyword plus dense) and reranking.
2. Every answer carries a citation that points back to the exact source chunk.
3. A 25-question golden set of question, ground-truth answer, and expected source lives in the repo.
4. A Ragas eval harness scores faithfulness, answer relevancy, and context recall, and the scores are committed as a file.
5. Langfuse tracing is wired in, so every request produces a span tree with per-step latency and cost.
6. A CI gate blocks the merge when any score falls below its threshold, demonstrated failing first, then passing.
7. Every model id and provider is in config, so the pipeline is provider-pluggable with no code change.
8. A README sells the project in thirty seconds, and a two-minute demo script walks a reviewer through it.

Prerequisites

This post pulls together earlier chapters, so skim any you are shaky on: RAG with LlamaIndex for the ingestion and querying basics, agentic RAG for hybrid search and reranking, LLM observability for Langfuse, and LLM-as-a-judge for the CI gate idea. Install the libraries into a fresh virtual environment: pip install llama-index chromadb ragas datasets langfuse. The project was built on Python 3.14.6 with llama-index 0.14.22, ragas 0.2.x, and langfuse 3.x, all current at the time of writing. The layout is a handful of small files: rag.config.json, pipeline.py, golden_set.json, eval.py, eval_gate.py, and a CI workflow.

Step 1: Provider-Pluggable Config

Model names change every few months, and a project that bakes them into the code rots fast. So we treat providers like a fuse box: every wire ends at a labeled switch in one panel, and you swap a component without rewiring the house. A single config holds the embedding model, the reranker, the generator, the vector store, and the eval thresholds. An environment variable can override any of it, which is handy in CI or when you want to try a cheaper model for one run. Here a developer named Aditi loads it and prints the resolved stack.

📄 config_demo.py: every model id in one place, env vars win

import json
import os

# Every model id and provider lives in config, never hardcoded in the pipeline.
# Swap providers by editing this file or setting an env var, no code change.
DEFAULT_CONFIG = {
    "embedding": {"provider": "openai", "model": "text-embedding-3-small"},
    "reranker":  {"provider": "cohere", "model": "rerank-3.5"},
    "generator": {"provider": "openai", "model": "gpt-5.4-mini"},
    "vector_store": {"provider": "chroma", "path": "./rag_index"},
    "thresholds": {"faithfulness": 0.90, "answer_relevancy": 0.85, "context_recall": 0.80},
}


def load_config(path="rag.config.json"):
    cfg = dict(DEFAULT_CONFIG)
    if os.path.exists(path):
        cfg.update(json.load(open(path, encoding="utf-8")))
    # An env var wins over the file, handy for CI and quick provider swaps.
    if os.getenv("RAG_GENERATOR_MODEL"):
        cfg["generator"] = {**cfg["generator"], "model": os.environ["RAG_GENERATOR_MODEL"]}
    return cfg


cfg = load_config()
print("Active stack (resolved from defaults + env):")
for part in ("embedding", "reranker", "generator", "vector_store"):
    print(f"  {part:<13} {cfg[part]['provider']:<8} {cfg[part].get('model', cfg[part].get('path'))}")

▶ Output

--- default run ---
Active stack (resolved from defaults + env):
  embedding     openai   text-embedding-3-small
  reranker      cohere   rerank-3.5
  generator     openai   gpt-5.4-mini
  vector_store  chroma   ./rag_index
--- with RAG_GENERATOR_MODEL override ---
Active stack (resolved from defaults + env):
  embedding     openai   text-embedding-3-small
  reranker      cohere   rerank-3.5
  generator     openai   claude-haiku-4.5
  vector_store  chroma   ./rag_index

What happened here: The first run resolved the stack from the defaults. The second run set RAG_GENERATOR_MODEL=claude-haiku-4.5 and the generator switched to a different provider’s model without touching a line of pipeline code. That is the whole trick behind dead-tech-proofing: the pipeline reads from config, so a model rename in six months is a one-line edit and a fresh eval run, not a rewrite. Notice the thresholds live here too, next to the models, because a stricter model deserves a stricter bar.

Step 2: Hybrid Retrieval with Citations

Two retrievers see the world differently. A keyword search is great when the question shares words with the document, like an index at the back of a book. A dense (embedding) search matches on meaning even when the words differ, like a helpful librarian who knows “paid time off” and “annual leave” are the same thing. Hybrid retrieval runs both and merges their results, so you get the best of the literal and the semantic. The clean way to merge two ranked lists is Reciprocal Rank Fusion (RRF): each document scores 1 / (k + rank) in each list, and the scores add up, so a document that both retrievers like floats to the top.

In the real RAG project the dense ranking comes from an embedding model and a vector store, which cost money to call. To keep this block runnable and honest, Anvay hands in the two ranked lists directly so you can watch the fusion math and the citation formatting, which is the part people get wrong.

📄 hybrid_retrieve.py: fuse two rankings, answer with a citation

# In the real app the dense ranking comes from an embedding model and a vector
# store; here we hand in two ranked lists so you can watch the fusion math.
CHUNKS = {
    "handbook.md#leave":  "Employees get 24 days of paid leave per year, accrued monthly.",
    "handbook.md#remote": "Remote work is allowed up to three days a week with manager sign-off.",
    "handbook.md#notice": "Notice period is 15 days during probation and 30 days after.",
    "menu.md#thali":      "The lunch thali has dal, paneer curry, rice, and two rotis.",
    "menu.md#snacks":     "Evening snacks rotate between samosa, dhokla, and masala corn.",
}


def rrf(ranked_lists, k=60):
    """Reciprocal Rank Fusion: blend several ranked lists into one score."""
    scores = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    fused = sorted(scores, key=scores.get, reverse=True)
    return fused, scores


# The two retrievers disagree on order; fusion reconciles them.
keyword_ranked = ["handbook.md#leave", "handbook.md#notice", "handbook.md#remote"]
dense_ranked   = ["handbook.md#leave", "handbook.md#remote", "handbook.md#thali"]

question = "How much paid leave do I get?"
fused, scores = rrf([keyword_ranked, dense_ranked])
top = fused[:3]

print(f"Q: {question}\n")
print("Fused ranking (reciprocal rank fusion of keyword + dense):")
for doc_id in top:
    print(f"  [{scores[doc_id]:.4f}] {doc_id}")

print("\nGrounded answer with a citation:")
print(f"  {CHUNKS[top[0]]} [1]")
print(f"  [1] {top[0]}")

▶ Output

Q: How much paid leave do I get?

Fused ranking (reciprocal rank fusion of keyword + dense):
  [0.0328] handbook.md#leave
  [0.0320] handbook.md#remote
  [0.0161] handbook.md#notice

Grounded answer with a citation:
  Employees get 24 days of paid leave per year, accrued monthly. [1]
  [1] handbook.md#leave

What happened here: Both retrievers ranked the leave chunk first, so RRF gave it the highest fused score and it won. The remote-work chunk placed third in one list and second in the other, so it came second overall; the notice chunk appeared only once, so it trailed. The answer then quotes the top chunk and tags it [1], with the citation resolving to the exact source id. That citation is the acceptance criterion that matters most, because it is what lets a user, or an interviewer, click through and confirm the answer is real instead of invented.

Step 3: The 25-Question Golden Set

You cannot grade a RAG project without an answer key. A golden set is that answer key: a fixed list of questions, the correct answer a human wrote for each, and the source chunk that should back it. Twenty-five is a sensible starting size, big enough to catch regressions, small enough to keep the eval cheap. Here is a slice of the file, then a validator, because a golden set with a broken row silently corrupts every score that depends on it.

📄 golden_set.json: a slice of the 25-question answer key

[
  {"id": 1, "question": "How many paid leave days do employees get?",
   "ground_truth": "24 days per year, accrued monthly.", "expected_source": "handbook.md#leave"},
  {"id": 2, "question": "How many days a week can I work remotely?",
   "ground_truth": "Up to three days a week with manager sign-off.", "expected_source": "handbook.md#remote"},
  {"id": 5, "question": "What is in the lunch thali?",
   "ground_truth": "Dal, paneer curry, rice, and two rotis.", "expected_source": "menu.md#thali"}
]

📄 validate_golden.py: refuse to ship a broken answer key

import json
import sys

REQUIRED = {"id", "question", "ground_truth", "expected_source"}
EXPECTED_COUNT = 25

golden = json.load(open("golden_set.json", encoding="utf-8"))

problems = []
seen_ids = set()
for row in golden:
    missing = REQUIRED - row.keys()
    if missing:
        problems.append(f"row {row.get('id', '?')} missing {missing}")
    if row["id"] in seen_ids:
        problems.append(f"duplicate id {row['id']}")
    seen_ids.add(row["id"])
    if not row["question"].strip().endswith("?"):
        problems.append(f"row {row['id']} question is not a question")

if len(golden) != EXPECTED_COUNT:
    problems.append(f"expected {EXPECTED_COUNT} rows, found {len(golden)}")

if problems:
    print("Golden set INVALID:")
    for p in problems:
        print(f"  - {p}")
    sys.exit(1)

print(f"Golden set OK: {len(golden)} questions, all keys present, ids unique.")

▶ Output (first run caught a bad row, second run after the fix)

Golden set INVALID:
  - row 19 question is not a question

Golden set OK: 25 questions, all keys present, ids unique.

What happened here: The first run failed and exited non-zero because row 19 read “Name one fried evening snack,” an instruction, not a question. That is exactly the kind of quiet mistake that skews a relevancy score without ever crashing. After rewording it to “Which fried item is an evening snack?” the validator passed: 25 rows, all keys present, ids unique. Run this validator in CI too, so a typo in the answer key can never sneak past and poison your eval numbers.

Step 4: The Ragas Eval Harness

Now the grading. Ragas is the tool most teams reach for to score a RAG system, and it measures three things that map cleanly to the questions a skeptic will ask. Faithfulness: is the answer actually supported by the retrieved context, or did the model wander off and invent? Answer relevancy: does the answer address the question that was asked? Context recall: did retrieval pull back the passages the ground-truth answer needs? Under the hood Ragas uses an LLM to judge each one, so this step needs an API key and a few cents per run, which is why the output below is marked as an example. The shape is exactly what you will see.

📄 eval.py: score the golden set and commit the numbers

import json

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_relevancy, context_recall, faithfulness

from pipeline import answer_question  # your hybrid pipeline from Step 2

golden = json.load(open("golden_set.json", encoding="utf-8"))

rows = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
for item in golden:
    result = answer_question(item["question"])   # returns answer + retrieved chunks
    rows["question"].append(item["question"])
    rows["answer"].append(result["answer"])
    rows["contexts"].append(result["contexts"])
    rows["ground_truth"].append(item["ground_truth"])

dataset = Dataset.from_dict(rows)
report = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_recall])

scores = {k: round(float(v), 3) for k, v in report.items()}
json.dump(scores, open("eval_scores.json", "w"), indent=2)
print("Wrote eval_scores.json:", scores)

▶ Example output

Evaluating: 100%|██████████████████████████| 25/25 [00:41<00:00,  1.65s/it]
Wrote eval_scores.json: {'faithfulness': 0.94, 'answer_relevancy': 0.91, 'context_recall': 0.86}

What happened here: This block is marked example output because Ragas calls an LLM judge for every row, which needs an API key the tutorial runner does not have. On a real run it walks all 25 golden questions, asks the pipeline for an answer and its retrieved contexts, and hands each to the judge. The three aggregate scores land in eval_scores.json, which you commit to the repo. That committed file is the point: it turns “the answers seemed fine” into a number a reviewer can trust and a gate can enforce. When you change a chunk size or swap a model, you rerun this and the numbers tell you if you made things better or worse.

Step 5: Langfuse Tracing

When an answer is wrong, you need to know which step failed: did retrieval miss, or did the model ignore good context? Tracing answers that. Langfuse records each request as a tree of spans, where every span has a name, a latency, and a cost, like an itemized receipt for a single question. You wire it in with a decorator, and the live SDK ships the tree to the Langfuse server, where you get one screenshot for the README. Here is the wiring, then a local script that prints a captured trace so you can see the exact shape without a server.

📄 pipeline.py: wrap each step in a Langfuse span

from langfuse import observe

@observe(name="retrieve.dense")
def dense_retrieve(question):
    ...   # embed the question, query the vector store

@observe(name="generate.llm")
def generate(question, contexts):
    ...   # call the generator with the retrieved context

@observe(name="rag.answer")          # the parent span for the whole request
def answer_question(question):
    contexts = hybrid_retrieve(question)
    answer = generate(question, contexts)
    return {"answer": answer, "contexts": contexts}

📄 trace_tree.py: print a captured trace locally (no server needed)

# A Langfuse trace is a tree of spans, each with a name, latency, and cost.
trace = {
    "name": "rag.answer", "latency_ms": 1840, "cost_usd": 0.0021,
    "spans": [
        {"name": "retrieve.keyword", "latency_ms": 40,  "cost_usd": 0.0},
        {"name": "retrieve.dense",   "latency_ms": 310, "cost_usd": 0.0004},
        {"name": "fuse.rrf",         "latency_ms": 2,   "cost_usd": 0.0},
        {"name": "rerank.cross",     "latency_ms": 180, "cost_usd": 0.0002},
        {"name": "generate.llm",     "latency_ms": 1305, "cost_usd": 0.0015},
    ],
}


def show(trace):
    print(f"{trace['name']:<20} {trace['latency_ms']:>5} ms  ${trace['cost_usd']:.4f}")
    for span in trace["spans"]:
        print(f"  {span['name']:<18} {span['latency_ms']:>5} ms  ${span['cost_usd']:.4f}")
    child_ms = sum(s["latency_ms"] for s in trace["spans"])
    slow = max(trace["spans"], key=lambda s: s["latency_ms"])
    print("-" * 44)
    print(f"  child spans total   {child_ms:>5} ms")
    print(f"  slowest span: {slow['name']} ({slow['latency_ms']} ms)")


show(trace)

▶ Output

rag.answer            1840 ms  $0.0021
  retrieve.keyword      40 ms  $0.0000
  retrieve.dense       310 ms  $0.0004
  fuse.rrf               2 ms  $0.0000
  rerank.cross         180 ms  $0.0002
  generate.llm        1305 ms  $0.0015
--------------------------------------------
  child spans total    1837 ms
  slowest span: generate.llm (1305 ms)

What happened here: The wiring block needs a live Langfuse key, so it is shown for shape. The trace printer, though, really ran, and it makes the value obvious: the generation step ate 1305 of the 1840 milliseconds, so if this request felt slow, the model call is where you look first, not retrieval. The per-span cost adds up to the request cost, which is how you spot a runaway prompt before the monthly bill does. In the real dashboard this same tree is clickable, and dropping one screenshot of it into your README is worth a paragraph of prose about “observability”.

Step 6: The CI Gate (Failing, Then Passing)

A committed score is nice, but a score nobody enforces drifts. The gate is what keeps the RAG project honest: a small script that reads eval_scores.json, compares each metric to its threshold, and exits non-zero if any fall short. CI reads that exit code and blocks the merge, the same way a coffee shop will not serve the next order until the card actually clears. We show it failing first, on the scores from a weaker early pipeline, then passing after retrieval was widened, so you can see the gate do its job.

📄 eval_gate.py: block the merge when quality drops

import json
import sys

# Thresholds the eval must clear before a merge is allowed. Tune per project.
THRESHOLDS = {
    "faithfulness": 0.90,      # answer is supported by the retrieved context
    "answer_relevancy": 0.85,  # answer actually addresses the question
    "context_recall": 0.80,    # retrieval pulled the passages the answer needs
}

scores = json.load(open(sys.argv[1], encoding="utf-8"))

print(f"Eval gate on {sys.argv[1]}")
print(f"{'metric':<18}{'score':>8}{'min':>8}   result")
print("-" * 46)
failed = []
for metric, floor in THRESHOLDS.items():
    got = scores.get(metric, 0.0)
    ok = got >= floor
    print(f"{metric:<18}{got:>8.3f}{floor:>8.2f}   {'PASS' if ok else 'FAIL'}")
    if not ok:
        failed.append(metric)

if failed:
    print(f"\nGATE FAILED on: {', '.join(failed)}. Merge blocked.")
    sys.exit(1)
print("\nGATE PASSED. All metrics clear their thresholds.")

▶ Output (run 1 = weak pipeline, run 2 = after widening retrieval)

===== RUN 1: before fixing retrieval =====
Eval gate on eval_scores_bad.json
metric               score     min   result
----------------------------------------------
faithfulness         0.710    0.90   FAIL
answer_relevancy     0.880    0.85   PASS
context_recall       0.620    0.80   FAIL

GATE FAILED on: faithfulness, context_recall. Merge blocked.
exit=1

===== RUN 2: after widening the retriever =====
Eval gate on eval_scores.json
metric               score     min   result
----------------------------------------------
faithfulness         0.940    0.90   PASS
answer_relevancy     0.910    0.85   PASS
context_recall       0.860    0.80   PASS

GATE PASSED. All metrics clear their thresholds.
exit=0

What happened here: Run 1 used the scores from an early version where retrieval was too narrow: faithfulness and context recall both fell below the floor, the gate printed the failing metrics, and exited 1, which blocks the pull request. After switching to hybrid retrieval with reranking, run 2 cleared every threshold and exited 0. Wire this into your CI workflow as one step, python eval_gate.py eval_scores.json, and quality stops being a promise and becomes a build status. A red gate is a feature, not an annoyance: it caught a real regression before a user did.

📄 .github/workflows/eval.yml: run the gate on every push

name: RAG Eval Gate

on:
  push:
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.14"
      - run: pip install -r requirements.txt
      - name: Validate the golden set
        run: python validate_golden.py
      - name: Score the golden set with Ragas
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python eval.py
      - name: Enforce the quality gate
        run: python eval_gate.py eval_scores.json

Step 7: The README and Demo Script

A repo with no README is a shop with the lights off. The README is the shop window: it tells a busy reviewer in thirty seconds what the RAG project does, that it is evaluated, and how to run it. This is interview exhibit A, so lead with the numbers and the trace, not adjectives.

📄 README.md: the template that sells the project

# Docs-QA: an evaluated RAG app

Ask questions over a folder of documents and get answers with citations.
Hybrid retrieval (keyword + dense) with reranking, scored by Ragas, traced
by Langfuse, and gated in CI so quality cannot regress silently.

![Eval Gate](https://github.com/YOUR_USER/docs-qa/actions/workflows/eval.yml/badge.svg)

## Quality (committed in eval_scores.json)
- Faithfulness: 0.94   - Answer relevancy: 0.91   - Context recall: 0.86
- Gate thresholds: 0.90 / 0.85 / 0.80  (merge blocked below any floor)

![Langfuse trace](docs/trace.png)

## Run it
    pip install -r requirements.txt
    python ingest.py ./my_docs      # build the index from your folder
    python ask.py "How much paid leave do I get?"

## Evaluate
    python validate_golden.py && python eval.py && python eval_gate.py eval_scores.json

## Stack (all model ids in rag.config.json, swap freely)
Embeddings, reranker, generator, vector store, Ragas, Langfuse.

The demo video is the other half. Recruiters skim, so a scripted two-minute tour of the RAG project beats a rambling ten. Here is the beat sheet Aviraj uses.

1. 0:00 to 0:20, the pitch. “This answers questions over your own documents, with citations, and it is evaluated so I can prove the answers are good.” Show the README.
2. 0:20 to 0:50, a real question. Run ask.py, read the answer aloud, then click the citation to the source chunk.
3. 0:50 to 1:20, the eval. Run the gate, point at the three scores and the thresholds, and say what each metric means in one line.
4. 1:20 to 1:45, the trace. Open the Langfuse trace, show the span tree, and note the slowest and priciest step.
5. 1:45 to 2:00, the gate in CI. Show a past pull request that went red on a low score, and say “this is why a bad change never reaches users.”

The Stack, and How to Swap Every Piece

Every component in this RAG project is a choice, not a commitment. This table lists what we used at the time of writing and a named alternative for each, so when a piece is renamed, priced out, or simply beaten, you swap it in config and rerun the eval. The eval is what makes swapping safe: change a component, rerun the golden set, and the numbers tell you if it was an upgrade.

Common Mistakes

• Shipping without an eval. “The answers looked fine” is not a measurement. A committed eval_scores.json and a gate turn opinion into evidence, and evidence is what an interviewer wants to see.
• Answers with no citation. An answer you cannot trace to a source is a confident guess. Always return the source chunk id so a reader can check, and so faithfulness has something to score against.
• A golden set nobody validates. One malformed row skews every score quietly. Validate the answer key in CI, as we did, before you trust a single number that depends on it.
• Hardcoding model names in the pipeline. Models get renamed and deprecated. Keep every id in config so a swap is one line and one eval run, not a refactor.
• Chasing a fancier model before fixing retrieval. If the right chunk never gets retrieved, no model can answer from it. When faithfulness is low, look at what was retrieved first, not at the generator.

Best Practices

• Commit the scores, not just the code. A tracked eval_scores.json lets a reviewer, and your future self, see quality at a glance and catch a regression in the diff.
• Make the gate fail the build. The eval step must exit non-zero below threshold, or CI is decoration. A green badge should mean the answers are still good.
• Trace every request from day one. When an answer is wrong at 2am, the span tree tells you whether retrieval or generation failed, in seconds instead of guesses.
• Keep the golden set honest and growing. When a real user hits a question the app gets wrong, add it to the golden set. Your eval gets smarter every time production surprises you.
• Put providers in config. One panel of switches means you can chase a cheaper or better model in an afternoon, backed by the eval, without touching the pipeline.

Conclusion

You just built a real RAG project end to end and ticked every acceptance box: hybrid retrieval with citations, a validated 25-question golden set, a Ragas eval harness whose scores are committed to the repo, Langfuse tracing that pinpoints the slow step, a CI gate demonstrated failing then passing, a README that leads with numbers, and a config where every model is a one-line swap. That combination, an app that answers well and proves it, is what separates a portfolio repo from a weekend demo.

Put this RAG project at the top of your resume, keep the eval gate green, and you have something concrete to walk through in any interview. For the full roadmap and every other topic in order, head to the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Shipping a RAG Project

If you can walk through these without peeking, you are ready for this topic in an interview.

Q: How do you prove your RAG app actually gives good answers?

I keep a golden set of 25 questions with human-written answers and expected sources, and I score the RAG project against it with Ragas on three metrics: faithfulness, answer relevancy, and context recall. The scores are committed to the repo and a CI gate blocks any merge that drops below threshold. So instead of saying the answers seem fine, I can point at a number that a robot re-checks on every push.

Q: Walk me through what happens when a question comes in.

The question fans out to two retrievers, a keyword one and a dense embedding one. Their ranked lists are merged with reciprocal rank fusion, then a reranker reorders the top candidates. The best passages go to the generator, which writes an answer and attaches a citation to the source chunk. Every step is a Langfuse span, so I can see latency and cost per stage afterward.

Q: Your faithfulness score dropped after a change. How do you find the cause?

Faithfulness low usually means the answer is not supported by what was retrieved, so I look at retrieval first. I open the trace for a failing question and check which chunks came back. If the right chunk is missing, the fix is in retrieval: chunk size, the embedding model, or how many chunks I fetch. If the right chunk is there but the answer ignored it, then it is a generation or prompting problem. The trace tells me which.

Q: Why fuse two retrievers instead of trusting the better one?

Because they fail on different questions. Keyword search wins on exact terms and codes, dense search wins on paraphrases and synonyms. Reciprocal rank fusion lets a passage that either retriever ranks highly still surface, so I get the strengths of both without having to pick. My eval confirmed it: context recall went up after moving from dense-only to hybrid, which is why the gate started passing.

Q: How is this project not going to be obsolete when the models change?

Every model id and provider lives in a config file, so a rename or a new model is a one-line change, not a rewrite. And because I have an eval, swapping a component is safe: I change it, rerun the golden set, and the scores tell me if it was an upgrade. The pipeline logic, retrieve, fuse, rerank, generate, evaluate, is stable even as the specific models underneath it churn.

Q: Scenario: a teammate named Anvi says the eval passes but users still complain. What do you do?

The golden set is missing the cases users actually hit. I collect the real failing questions, add them to the golden set with correct answers, and rerun the eval; the scores will likely drop, which is good, because now the gate reflects reality. Then I fix retrieval or prompting until it passes again. A golden set is a living thing: every production surprise becomes a new test so the same miss cannot happen twice.

Reference: the complete, always-current details live in the official Python documentation.

Related Posts

Previous: Agentic RAG in Python: Hybrid Search, Reranking, and GraphRAG

Next: GenAI: Fine-Tuning LLMs with Low-Rank Adaptation (LoRA), QLoRA, and Parameter-Efficient Fine-Tuning (PEFT)

Series Home: Python + AI/ML Tutorial Series