Plain RAG works right up until real users show up: typed keywords miss, and questions that span two files come back half answered. Agentic RAG treats retrieval as a loop the system can steer, rewriting the query, blending keyword and vector search, reranking, and retrieving again. This post upgrades the LlamaIndex pipeline with real numbers for every step, so you tune with proof instead of vibes.
One quick definition first, because “agentic” leans on an idea we have not properly met yet. A plain model answers your question once and then stops. An AI agent is an LLM handed a goal plus a set of tools, and it works in a loop: it thinks about what to do next, takes an action (like running a search or calling a function), looks at the result, and decides again, repeating until the goal is met. So “agentic retrieval” is just that idea pointed at search: you let the model decide what to look up and when, instead of always fetching once and hoping the first pull was enough. We build a full agent from scratch in the AI agents tutorial later in this series; for now the loop-until-done picture is all you need.
“If you cannot measure your retrieval, you are not tuning it, you are guessing.”
Last Updated: July 2026 | Tested on: Python 3.14.6, sentence-transformers 5.6, networkx 3.6 | Difficulty: Expert | Reading Time: 27 minutes
- RAG with LlamaIndex (baseline pipeline this post builds on)
- LLM tool calling (the retrieve-then-reason loop reused here)
- Trees and graphs (the payoff behind GraphRAG)
- pip install sentence-transformers networkx. The BM25 code is pure standard library.
Table of Contents
Beyond Baseline RAG
Read a few 2026 job postings for GenAI (Generative AI) engineers and you will notice they never just say “knows RAG.” They ask for hybrid search, reranking, evaluation, and agentic retrieval. That is because baseline RAG is now table stakes, and the interesting problems all live one layer up. Here is the honest map of what “advanced” actually means, and which real pain each piece solves.
- Chunking you can measure: stop guessing chunk size, tie it to a recall number on a fixed eval set.
- Hybrid search: combine keyword (BM25) and semantic (vector) retrieval so neither blind spot sinks you.
- Reranking: a second, slower model reorders the shortlist so the best chunk lands at the top.
- Agentic retrieval: rewrite the query and retrieve again in a loop, instead of one static lookup.
- GraphRAG: answer questions whose answer is a chain of relationships, not a single passage.
We will walk each one with a running example: GreenBowl, a fictional vegetarian meal-kit service whose help docs we want to search. Small corpus, real code, real output.
The Advanced RAG Pipeline
Before the code, here is the whole system in one picture. Baseline RAG is the straight line embed, retrieve, generate. The advanced pipeline adds decision points: a router that picks the retrieval strategy, a fusion step, a reranker, and a loop back to the top when the evidence is thin.
Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.
Notice the two decisions. The first, near the top, routes a query: a plain factual question goes through hybrid search, while a multi-hop relational question (“which chef made a dish using cheese”) goes through GraphRAG. The second, near the bottom, is the agentic part: if the reranked, permission-filtered evidence is not enough to answer, the system rewrites the query and goes around again. That loop is the same ask-check-act cycle from the tool calling tutorial, pointed at retrieval instead of tools.
Chunking: Measure Recall, Do Not Guess
Most teams pick a chunk size because a blog post told them 512 tokens, then never check it. That is backwards. Chunking is the cheapest lever in RAG and the easiest to measure. Think of it like slicing bread for sandwiches: cut the slices too thin and each one falls apart with nothing on it, cut them too thick and one slice hogs the whole loaf. You want slices sized to the job, and the only way to know is to weigh the result.
So we measure. Here a developer named Aditi takes the GreenBowl help doc, chunks it at four different sizes, and runs a fixed set of six questions through a plain BM25 retriever. For each chunk size she records Recall@3 (did the sentence that answers the question land in the top three chunks) and how many total sentences got stuffed into the prompt. The BM25 here is about forty lines of standard-library Python, no external search engine needed.
📄 chunk_recall.py: tie chunk size to a recall number
import re, math
from collections import Counter
# GreenBowl help doc, one sentence per line. This is our knowledge base.
DOC = """GreenBowl delivers vegetarian meal kits to your door every week.
You can pause or skip a delivery any time before the Thursday 6 PM cutoff.
Refunds for a spoiled box are issued within 14 days of the delivery date.
To report a spoiled box, open the app and tap Help then Report an Issue.
Each recipe card lists the exact cooking time and the calorie count.
The Paneer Tikka kit takes 25 minutes and serves two people.
Our API rate limit is 60 requests per minute per API key.
If you exceed the rate limit the API returns HTTP status 429.
API keys are created in the developer dashboard under Settings then Keys.
A leaked API key should be revoked immediately from the same Keys page.
The mobile app supports both light mode and dark mode themes.
Delivery is free on orders above 40 dollars, otherwise it costs 5 dollars.
We deliver to all metro pin codes but not yet to remote rural areas.
The loyalty program gives you 1 point for every 10 dollars you spend.
Points never expire and 100 points convert into a 5 dollar credit.
Gift cards are non refundable and cannot be exchanged for cash.""".strip().split("\n")
# Eval set: (question, index of the one sentence that answers it). Hand-labelled.
QUERIES = [
("how long do I have to get money back for a bad box", 2),
("what happens when I send too many API calls", 7),
("where do I make a new API key", 8),
("is shipping free", 11),
("how do loyalty points turn into credit", 14),
("can I get cash for a gift card", 15),
]
def tok(t): return re.findall(r"[a-z0-9]+", t.lower())
class BM25: # a compact, dependency-free BM25
def __init__(self, docs, k1=1.5, b=0.75):
self.d=[tok(x) for x in docs]; self.k1=k1; self.b=b
self.N=len(self.d); self.avg=sum(len(x) for x in self.d)/self.N
df=Counter()
for x in self.d:
for w in set(x): df[w]+=1
self.idf={w: math.log(1+(self.N-n+0.5)/(n+0.5)) for w,n in df.items()}
self.tf=[Counter(x) for x in self.d]
def search(self, q, k):
qt=tok(q); out=[]
for i,tf in enumerate(self.tf):
dl=len(self.d[i]); s=0.0
for w in qt:
if w in tf:
s+=self.idf.get(w,0)*tf[w]*(self.k1+1)/(tf[w]+self.k1*(1-self.b+self.b*dl/self.avg))
out.append((i,s))
return [i for i,_ in sorted(out, key=lambda z:-z[1])[:k]]
def chunks_of(sentences, group, overlap):
step=max(1, group-overlap); out=[]; i=0
while i < len(sentences):
idx=list(range(i, min(i+group, len(sentences))))
out.append((" ".join(sentences[j] for j in idx), set(idx)))
if i+group >= len(sentences): break
i+=step
return out
print(f"Corpus: {len(DOC)} sentences | {len(QUERIES)} queries | retrieving top-3 chunks\n")
print(f"{'chunk (sentences)':<19}{'#chunks':<9}{'Recall@3':<11}{'sentences in prompt':<21}{'wasted':<8}")
for group in (1, 2, 3, 6):
ch = chunks_of(DOC, group, overlap=1 if group > 1 else 0)
bm = BM25([c[0] for c in ch])
hits=0; ctx=0
for q, gold in QUERIES:
got=set()
for ci in bm.search(q, 3): got |= ch[ci][1]
hits += (gold in got); ctx += len(got)
recall=hits/len(QUERIES); avg_ctx=ctx/len(QUERIES); wasted=100*(1-len(QUERIES)/ctx)
print(f"{group:<19}{len(ch):<9}{recall:<11.2f}{avg_ctx:<21.1f}{wasted:<8.0f}%")
▶ Output
Corpus: 16 sentences | 6 queries | retrieving top-3 chunks chunk (sentences) #chunks Recall@3 sentences in prompt wasted 1 16 0.83 3.0 67 % 2 15 1.00 4.5 78 % 3 8 1.00 7.3 86 % 6 3 1.00 16.0 94 %
What happened here: One-sentence chunks missed an answer entirely, Recall@3 sat at 0.83, because a single short sentence rarely carries enough matching words for BM25 to rank it in the top three. Bump to two or three sentences and recall hits 1.00. Keep going to six-sentence chunks and recall stays 1.00, but look at the last two columns: the prompt now holds all 16 sentences and 94 percent of them are noise the model has to read past. That is the real tradeoff.
Bigger chunks help recall until they start dumping the whole corpus into every prompt, which raises token cost and gives the model more room to get distracted. The knee here is two to three sentences. Your corpus will have a different knee, and the only way to find it is to run this table on your own data.
Hybrid Search: BM25 Plus Vectors with RRF
Vector search is great at meaning and blind to exact words. Keyword search is the opposite. Picture looking for a book in a library: semantic search is asking a helpful librarian “something about turning points into store credit,” while keyword search is scanning the spines for the exact title. Sometimes you know the phrase, sometimes you only know the idea. Hybrid search runs both and merges the results, so a rare error code and a fuzzy paraphrase both find their answer.
The clean way to merge two ranked lists is Reciprocal Rank Fusion (RRF). It ignores the raw scores, which live on totally different scales, and only looks at each document’s rank in each list, adding up 1/(k+rank). Here a developer named Anvay runs the same three queries through BM25, a real vector model, and RRF, and prints where the correct answer landed in each.
📄 hybrid_rrf.py: fuse keyword and semantic retrieval
import os, warnings, re, math
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"]="1"; os.environ["TRANSFORMERS_VERBOSITY"]="error"
warnings.filterwarnings("ignore")
from collections import Counter
from sentence_transformers import SentenceTransformer
import numpy as np
PASSAGES = [
"Refunds for a spoiled box are issued within 14 days of the delivery date.", #0
"You can pause or skip a delivery before the Thursday 6 PM cutoff.", #1
"If you exceed the rate limit the API returns HTTP status 429.", #2
"API keys are created in the developer dashboard under Settings then Keys.", #3
"Delivery is free on orders above 40 dollars, otherwise it costs 5 dollars.", #4
"Points never expire and 100 points convert into a 5 dollar credit.", #5
"The Paneer Tikka kit takes 25 minutes and serves two people.", #6
"Gift cards are non refundable and cannot be exchanged for cash.", #7
]
def tok(t): return re.findall(r"[a-z0-9]+", t.lower())
class BM25:
def __init__(s, docs, k1=1.5, b=0.75):
s.d=[tok(x) for x in docs]; s.k1=k1; s.b=b; s.N=len(s.d); s.avg=sum(len(x) for x in s.d)/s.N
df=Counter()
for x in s.d:
for w in set(x): df[w]+=1
s.idf={w: math.log(1+(s.N-n+0.5)/(n+0.5)) for w,n in df.items()}; s.tf=[Counter(x) for x in s.d]
def rank(s, q):
qt=tok(q); out=[]
for i,tf in enumerate(s.tf):
dl=len(s.d[i]); sc=0.0
for w in qt:
if w in tf: sc+=s.idf.get(w,0)*tf[w]*(s.k1+1)/(tf[w]+s.k1*(1-s.b+s.b*dl/s.avg))
out.append((i,sc))
return [i for i,_ in sorted(out, key=lambda z:-z[1])]
# Reciprocal Rank Fusion: blend two ranked lists by 1/(k+rank), no score scaling needed.
def rrf(rankings, k=60):
score=Counter()
for r in rankings:
for pos, doc in enumerate(r): score[doc]+=1.0/(k+pos+1)
return [d for d,_ in score.most_common()]
model=SentenceTransformer("all-MiniLM-L6-v2")
emb=model.encode(PASSAGES, normalize_embeddings=True)
def dense_rank(q):
qv=model.encode([q], normalize_embeddings=True)[0]
return [int(i) for i in np.argsort(-(emb@qv))]
bm=BM25(PASSAGES)
tests = [
("status code 429 meaning", 2), # exact rare token
("vegetarian cheese dish cooking time", 6), # zero shared words with 'paneer'
("how do I get my money back for a bad delivery", 0), # paraphrase, shares 'delivery'
]
def pos(lst, g): return lst.index(g)+1
for q, gold in tests:
kw=bm.rank(q); dn=dense_rank(q); fs=rrf([kw, dn])
print(f"Query: {q!r} (answer = passage #{gold})")
print(f" BM25 rank of answer = {pos(kw,gold)} top-3 = {kw[:3]}")
print(f" Dense rank of answer = {pos(dn,gold)} top-3 = {dn[:3]}")
print(f" RRF rank of answer = {pos(fs,gold)} top-3 = {fs[:3]}")
print()
▶ Output
Query: 'status code 429 meaning' (answer = passage #2) BM25 rank of answer = 1 top-3 = [2, 0, 1] Dense rank of answer = 1 top-3 = [2, 3, 1] RRF rank of answer = 1 top-3 = [2, 3, 1] Query: 'vegetarian cheese dish cooking time' (answer = passage #6) BM25 rank of answer = 7 top-3 = [0, 1, 2] Dense rank of answer = 1 top-3 = [6, 1, 2] RRF rank of answer = 4 top-3 = [1, 2, 0] Query: 'how do I get my money back for a bad delivery' (answer = passage #0) BM25 rank of answer = 1 top-3 = [0, 1, 7] Dense rank of answer = 3 top-3 = [4, 1, 0] RRF rank of answer = 1 top-3 = [0, 1, 4]
What happened here: three queries, three different lessons. On “status code 429” the exact token is in the passage, so both methods nail it at rank 1. On “vegetarian cheese dish,” BM25 falls to rank 7 because the passage says “Paneer Tikka” and shares zero words with the query, while the vector model knows paneer is a cheese and puts it first. On “money back for a bad delivery,” it flips: BM25 gets it at rank 1 while the vector model is distracted by the “free delivery” passage and drops the refund answer to rank 3. That is exactly when keyword beats semantic, shared surface words that the embedding smooths over.
RRF is not magic, on the cheese query it still lands the answer at rank 4 because one retriever whiffed badly, but across the set it is the most reliable single column. In production you fuse both and let the reranker below clean up the top.
Reranking with a Cross-Encoder
Retrieval is a two-speed game. The first stage (BM25, vectors, RRF) is fast and casts a wide net, pulling maybe the top 20 or 50 candidates. It is cheap because it compares pre-computed vectors. A reranker is the slow, careful second stage: a cross-encoder reads the query and each candidate together and scores how well they actually match. It is like a first-round resume screen followed by a real interview. The screen is quick and rough, the interview is expensive but far more accurate, so you only interview the shortlist.
Here a developer named Aviraj takes a BM25 ranking, measures its nDCG@3 against hand-labelled relevance grades, then reranks the same candidates with a cross-encoder and measures again. nDCG rewards putting the most relevant passages highest, so a jump means the good chunks moved up.
📄 rerank_ndcg.py: measure the quality lift from a cross-encoder
import os, warnings, re, math, time
os.environ["HF_HUB_DISABLE_SYMLINKS_WARNING"]="1"; os.environ["TRANSFORMERS_VERBOSITY"]="error"
warnings.filterwarnings("ignore")
from collections import Counter
from sentence_transformers import CrossEncoder
PASSAGES = [
"The Paneer Tikka kit takes 25 minutes and serves two people.", #0
"Each recipe card lists the exact cooking time and the calorie count.", #1
"GreenBowl delivers vegetarian meal kits to your door every week.", #2
"The Chana Masala kit takes 30 minutes and serves four people.", #3
"You can pause or skip a delivery before the Thursday 6 PM cutoff.", #4
"Refunds for a spoiled box are issued within 14 days of delivery.", #5
]
# Graded relevance for the query below (0=irrelevant, 3=perfect). Human-labelled.
QUERY = "how many minutes to cook the paneer kit"
REL = {0:3, 1:2, 3:1, 2:0, 4:0, 5:0}
def tok(t): return re.findall(r"[a-z0-9]+", t.lower())
class BM25:
def __init__(s, docs, k1=1.5, b=0.75):
s.d=[tok(x) for x in docs]; s.k1=k1; s.b=b; s.N=len(s.d); s.avg=sum(len(x) for x in s.d)/s.N
df=Counter()
for x in s.d:
for w in set(x): df[w]+=1
s.idf={w:math.log(1+(s.N-n+0.5)/(n+0.5)) for w,n in df.items()}; s.tf=[Counter(x) for x in s.d]
def rank(s, q):
qt=tok(q); out=[]
for i,tf in enumerate(s.tf):
dl=len(s.d[i]); sc=0.0
for w in qt:
if w in tf: sc+=s.idf.get(w,0)*tf[w]*(s.k1+1)/(tf[w]+s.k1*(1-s.b+s.b*dl/s.avg))
out.append((i,sc))
return [i for i,_ in sorted(out, key=lambda z:-z[1])]
def dcg(order): return sum((2**REL[d]-1)/math.log2(pos+2) for pos,d in enumerate(order))
def ndcg(order, k):
ideal=sorted(REL, key=lambda d:-REL[d])[:k]
return dcg(order[:k])/dcg(ideal) if dcg(ideal) else 0.0
bm=BM25(PASSAGES)
first_stage=bm.rank(QUERY) # BM25 ranking (fast, cheap)
print(f"Query: {QUERY!r}\n")
print(f"BM25 order : {first_stage}")
print(f"nDCG@3 before : {ndcg(first_stage,3):.3f}")
ce=CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
t0=time.perf_counter()
pairs=[(QUERY, PASSAGES[i]) for i in first_stage]
scores=ce.predict(pairs)
reranked=[i for _,i in sorted(zip(scores, first_stage), key=lambda z:-z[0])]
dt=(time.perf_counter()-t0)*1000
print(f"Reranked order : {reranked}")
print(f"nDCG@3 after : {ndcg(reranked,3):.3f}")
print(f"Rerank latency : {dt:.0f} ms for {len(pairs)} candidates")
▶ Output
Query: 'how many minutes to cook the paneer kit' BM25 order : [0, 3, 2, 1, 4, 5] nDCG@3 before : 0.812 Reranked order : [0, 3, 1, 2, 5, 4] nDCG@3 after : 0.972 Rerank latency : 33 ms for 6 candidates
What happened here: BM25 ranked the recipe-card passage (relevance 2) below a totally irrelevant “GreenBowl delivers meal kits” line, because the irrelevant line happened to share the word “kit.” nDCG@3 was 0.812. The cross-encoder read query and passage together, understood the recipe card was more on-topic, and swapped them, lifting nDCG@3 to 0.972. That is a real 16 percent relative jump from a component you bolt on in five lines. The cost is latency: 33 milliseconds here for six candidates on a laptop Central Processing Unit (CPU), and it scales with the shortlist length, which is exactly why you rerank the top 20 or 50 and never the whole corpus. First stage for recall, reranker for precision.
Agentic RAG: Rewrite, Retrieve, Reason, Repeat
Everything so far still does one retrieval per question. Agentic RAG breaks that assumption. Some questions cannot be answered by any single chunk, because the answer is two facts you have to chain. “How much must I spend to earn a loyalty credit?” needs the conversion rate (100 points equal a 5 dollar credit) and the earning rate (1 point per 10 dollars), and those live in different sentences. A human would look up one, realise a piece is missing, and look up the other. That is the loop.
Here a developer named Anvi wires up that retrieve-then-reason loop. To keep it runnable offline, the three “thinking” helpers (decide if evidence is complete, rewrite the query, write the final answer) are plain Python rules. In a real build each of those is an LLM call, but the control flow is identical, and it is the same guarded loop from the tool-calling post.
📄 agentic_loop.py: multi-hop retrieve, reason, repeat
import re, math
from collections import Counter
KB = [
"GreenBowl delivers vegetarian meal kits to your door every week.",
"The loyalty program gives you 1 point for every 10 dollars you spend.",
"Points never expire and 100 points convert into a 5 dollar credit.",
"Refunds for a spoiled box are issued within 14 days of delivery.",
"Delivery is free on orders above 40 dollars, otherwise it costs 5 dollars.",
"The Paneer Tikka kit takes 25 minutes and serves two people.",
]
STOP = {"how","do","i","to","get","a","the","you","of","for","and","is","in","need","much","are","on"}
def tok(t): return [w for w in re.findall(r"[a-z0-9]+", t.lower()) if w not in STOP]
class BM25:
def __init__(s, docs):
s.d=[tok(x) for x in docs]; s.N=len(s.d); s.avg=sum(len(x) for x in s.d)/s.N
df=Counter()
for x in s.d:
for w in set(x): df[w]+=1
s.idf={w:math.log(1+(s.N-n+0.5)/(n+0.5)) for w,n in df.items()}; s.tf=[Counter(x) for x in s.d]
def top(s, q, k=1):
qt=tok(q); out=[]
for i,tf in enumerate(s.tf):
dl=len(s.d[i]); sc=0.0
for w in qt:
if w in tf: sc+=s.idf.get(w,0)*tf[w]*2.5/(tf[w]+1.5*(0.25+0.75*dl/s.avg))
out.append((i,sc))
out.sort(key=lambda z:-z[1]); return out[:k]
retriever = BM25(KB)
# These three helpers stand in for LLM calls so the loop runs offline.
# In production each one is a prompt: a router, a query rewriter, a synthesiser.
def has_conversion(text): return "convert into" in text
def has_rate(text): return "every 10 dollars" in text
def is_complete(text): return has_conversion(text) and has_rate(text)
def rewrite(evidence_text):
if has_conversion(evidence_text) and not has_rate(evidence_text):
return "how are loyalty points earned per dollar spent"
return "loyalty points credit"
def synthesise(evidence):
text = " ".join(evidence)
if is_complete(text):
return ("100 points convert to a 5 dollar credit, and you earn 1 point per 10 "
"dollars, so you must spend 1000 dollars to reach one credit.")
return "Not enough information retrieved."
question = "how much must I spend to earn a loyalty credit"
query = "how do I earn a loyalty credit"
evidence, seen = [], set()
for hop in range(1, 4):
idx, score = retriever.top(query, k=1)[0]
passage = KB[idx]
print(f"Hop {hop}: query={query!r}")
print(f" retrieved [{score:.2f}] -> {passage}")
if idx not in seen:
evidence.append(passage); seen.add(idx)
if is_complete(" ".join(evidence)):
print(f" router: all facts gathered, exit loop\n"); break
query = rewrite(" ".join(evidence))
print(f" router: missing a fact, rewrite and retrieve again\n")
print("Final answer:")
print(" ", synthesise(evidence))
▶ Output
Hop 1: query='how do I earn a loyalty credit'
retrieved [1.57] -> The loyalty program gives you 1 point for every 10 dollars you spend.
router: missing a fact, rewrite and retrieve again
Hop 2: query='loyalty points credit'
retrieved [3.64] -> Points never expire and 100 points convert into a 5 dollar credit.
router: all facts gathered, exit loop
Final answer:
100 points convert to a 5 dollar credit, and you earn 1 point per 10 dollars, so you must spend 1000 dollars to reach one credit.
What happened here: the first hop pulled the earning rate but the router saw the conversion fact was still missing, so it rewrote the query and retrieved again. The second hop filled the gap, the router declared the evidence complete, and the loop exited before burning a wasted third hop. A baseline one-shot retriever would have grabbed one sentence and answered half the question. Two things make this safe: the range-limited loop (never more than three hops here) so a confused agent cannot spin forever, and the completeness check so it stops the moment it has what it needs. Swap the rule-based helpers for LLM prompts and you have production agentic RAG. The scaffolding does not change.
GraphRAG: When Answers Live in the Connections
Some questions are not about any passage at all, they are about relationships. “Which chefs created a dish that uses a cheese, and where do they work?” No single sentence holds that answer. You have to chain: cheese is a category, paneer is a cheese, some dish uses paneer, some chef created that dish, and that chef works somewhere. Vector search struggles here because each hop is a separate fact and similarity cannot follow the chain. This is where the graphs you learned earlier pay off. GraphRAG stores your knowledge as a graph of entities and relationships, then answers by walking edges.
Here a developer named Aditi builds a tiny knowledge graph of chefs, dishes, and ingredients with networkx, then answers the multi-hop question by traversing it. Real data teams build these graphs automatically by having an LLM extract triples from documents, but the retrieval step is exactly this walk.
📄 graph_rag.py: answer a multi-hop question by walking a graph
import networkx as nx
# A tiny knowledge graph: (subject) -[relation]-> (object)
TRIPLES = [
("Aditi", "created", "Paneer Tikka"),
("Anvay", "created", "Chana Masala"),
("Aviraj", "created", "Palak Paneer"),
("Paneer Tikka", "uses", "Paneer"),
("Palak Paneer", "uses", "Paneer"),
("Palak Paneer", "uses", "Spinach"),
("Chana Masala", "uses", "Chickpeas"),
("Paneer", "is_a", "Cheese"),
("Aditi", "works_in", "Bangalore Kitchen"),
("Aviraj", "works_in", "Bangalore Kitchen"),
("Anvay", "works_in", "Pune Kitchen"),
]
G = nx.DiGraph()
for s, r, o in TRIPLES:
G.add_edge(s, o, relation=r)
print(f"Graph: {G.number_of_nodes()} nodes, {G.number_of_edges()} edges\n")
def parents(node, rel):
return [u for u, v, d in G.in_edges(node, data=True) if d["relation"] == rel]
def children(node, rel):
return [v for u, v, d in G.out_edges(node, data=True) if d["relation"] == rel]
# Walk: Cheese <-is_a- ingredient <-uses- dish <-created- chef -works_in-> kitchen
answers = []
for ingredient in parents("Cheese", "is_a"):
for dish in parents(ingredient, "uses"):
for chef in parents(dish, "created"):
for kitchen in children(chef, "works_in"):
answers.append((chef, dish, ingredient, kitchen))
print("Question: which chefs created a dish that uses a cheese, and where do they work?\n")
for chef, dish, ing, kit in sorted(answers):
print(f" {chef} -> created {dish} -> uses {ing} (a cheese) -> works in {kit}")
▶ Output
Graph: 12 nodes, 11 edges Question: which chefs created a dish that uses a cheese, and where do they work? Aditi -> created Paneer Tikka -> uses Paneer (a cheese) -> works in Bangalore Kitchen Aviraj -> created Palak Paneer -> uses Paneer (a cheese) -> works in Bangalore Kitchen
What happened here: the walk followed four relationship hops and returned both chefs whose dishes use a cheese, along with their kitchens, with the full reasoning path attached so a human can audit it. Anvay was correctly left out, chana masala uses chickpeas, not cheese. A pure vector store cannot do this reliably because “cheese” and “Bangalore Kitchen” never appear near each other in any single chunk. GraphRAG shines exactly when the answer is a path, not a paragraph: org charts, supply chains, dependency trees, medical relationships. The catch is you need the graph, and building a clean one from messy documents is the hard part of GraphRAG, not the traversal.
ACL-Aware Retrieval and the Managed-RAG Question
One rule matters more than every accuracy trick combined: never retrieve a chunk the user is not allowed to read. If the finance report is in your vector store and a marketing user asks a question that happens to match it, a naive RAG app will paste that report straight into the prompt and the model will happily summarise it. The fix is to filter by access control (ACL) before ranking, never after. Here a developer named Anvay tags each chunk with the groups allowed to see it and filters on the user’s groups first.
📄 acl_retrieval.py: permission filtering before ranking
# Each chunk carries the groups allowed to see it. Filter BEFORE ranking,
# never after: an unauthorised chunk must never reach the LLM prompt.
CHUNKS = [
{"text": "GreenBowl public menu: Paneer Tikka, Chana Masala, Palak Paneer.", "acl": {"public"}},
{"text": "Q3 revenue was 4.2 crore, up 18 percent over Q2.", "acl": {"finance"}},
{"text": "Supplier contract with FreshFarms renews in March 2027.", "acl": {"ops", "finance"}},
{"text": "Delivery is free on orders above 40 dollars.", "acl": {"public"}},
]
def retrieve(query_terms, user_groups, k=5):
allowed = [c for c in CHUNKS if c["acl"] & user_groups] # ACL filter first
scored = [(sum(t in c["text"].lower() for t in query_terms), c) for c in allowed]
scored = [(s, c) for s, c in scored if s > 0]
scored.sort(key=lambda z: -z[0])
return [c["text"] for _, c in scored[:k]]
q = ["revenue", "menu", "delivery"]
print("Marketing user (groups: public):")
for t in retrieve(q, {"public"}): print(" ", t)
print("\nFinance user (groups: public, finance):")
for t in retrieve(q, {"public", "finance"}): print(" ", t)
▶ Output
Marketing user (groups: public):
GreenBowl public menu: Paneer Tikka, Chana Masala, Palak Paneer.
Delivery is free on orders above 40 dollars.
Finance user (groups: public, finance):
GreenBowl public menu: Paneer Tikka, Chana Masala, Palak Paneer.
Q3 revenue was 4.2 crore, up 18 percent over Q2.
Delivery is free on orders above 40 dollars.
What happened here: the same query returned different results per user. The marketing user never even saw the revenue chunk exist, because the ACL filter dropped it before scoring. The finance user got it. This is the single most important thing to get right in an enterprise RAG app, and it is the reason a lot of teams reach for a managed service instead of wiring it all by hand.
Common Mistakes
- Tuning without an eval set: changing chunk size or the embedding model and eyeballing one answer tells you nothing. Build a small labelled query set and track Recall@K and nDCG, like the tables above.
- Vector-only retrieval: pure semantic search quietly fails on exact terms, product codes, and error numbers. Add BM25 and fuse with RRF before you blame the model.
- Reranking the whole corpus: a cross-encoder is slow. Rerank the top 20 to 50 candidates, never all of them, or latency explodes.
- Unbounded agentic loops: a retrieve-then-reason loop with no hop limit can spin forever and run up a bill. Always cap the hops.
- Filtering permissions after retrieval: if an unauthorised chunk enters the prompt at all, it has leaked. Filter by ACL before ranking, always.
Best Practices
- Measure first, upgrade second: add each component (hybrid, rerank, agentic) only after a number proves the previous stage was the bottleneck.
- Two-stage retrieval by default: a fast, wide first stage for recall, then a cross-encoder reranker for precision, is the reliable production shape.
- Route by query type: send factual questions to hybrid search and multi-hop relational questions to GraphRAG, rather than forcing one strategy on everything.
- Keep techniques, swap tools freely: chunking, RRF, reranking, and traversal outlive any one library. LlamaIndex, Chroma, and pgvector are current at the time of writing, and Qdrant or Weaviate are equally valid vector stores.
- Cite everything: carry the source of every retrieved chunk through to the answer so users and auditors can check it.
Conclusion
Agentic RAG is not one big idea, it is a stack of measurable upgrades over the baseline pipeline. You saw chunk size move Recall@3 with the tradeoff of prompt noise, hybrid search rescue the queries that vectors or keywords alone fumbled, a cross-encoder lift nDCG@3 from 0.812 to 0.972, a retrieve-then-reason loop chain two facts a one-shot retriever would miss, and a knowledge graph answer a question that has no single passage.
Underneath the frameworks, these are the durable skills. Tools like LlamaIndex, Chroma, and pgvector will keep changing, but Recall@K, RRF, reranking, and graph traversal are the layer that lasts. The one habit that ties it together is measurement: never upgrade a RAG system on a hunch, upgrade it because a number told you where it hurt.
Want the full path from Python basics to production AI? Explore the complete Python + AI/ML tutorial series home.
Frequently Asked Questions
What is the difference between agentic RAG and normal RAG?
Normal RAG does one retrieval per question: embed the query, pull the top-K chunks, generate an answer. Agentic RAG treats retrieval as a loop the system can steer. It can rewrite a weak query, route to a different retrieval strategy, retrieve more than once to chain facts, and stop only when it has enough evidence. The building blocks are the same, the difference is the control loop around them.
Is hybrid search worth the extra complexity over pure vector search?
Usually yes. Pure vector search quietly fails on exact tokens like product codes, error numbers, and rare names, because embeddings smooth those away. BM25 catches them. Fusing the two with Reciprocal Rank Fusion adds only a few lines and no new infrastructure if your store supports both, and it removes a whole class of silent retrieval misses. Measure it on your own eval set before deciding.
Do rerankers make RAG too slow?
Only if you rerank too much. A cross-encoder scores the query against each candidate together, so it is far slower per item than comparing pre-computed vectors. The fix is the two-stage pattern: retrieve a wide shortlist cheaply, then rerank just the top 20 to 50. On that shortlist the added latency is usually tens of milliseconds, a fair price for the precision gain.
When should I use GraphRAG instead of vector RAG?
Use GraphRAG when answers are relationships rather than passages: multi-hop questions across org charts, supply chains, dependencies, or linked entities. Vector search cannot reliably chain facts that never appear in the same chunk. Use plain vector or hybrid RAG when answers live inside individual passages, which is most factual question answering. Many production systems route between the two based on the question.
Interview Questions on Agentic RAG
Scenario questions, not trivia: this is the form this topic takes in a real interview.
Q: Your RAG app returns confident but wrong answers even though the right facts are in the corpus. How do you debug it?
Check retrieval before the model. Build a small labelled eval set and measure Recall@K: is the correct chunk even in the top-K? If not, the bug is retrieval, so revisit chunk size, add hybrid search, or add a reranker. If the right chunk is retrieved but ranked low, a cross-encoder reranker usually fixes it. Only after retrieval is solid do you look at the generation prompt.
Q: Explain Reciprocal Rank Fusion and why it beats just averaging two similarity scores.
RRF combines ranked lists using 1/(k+rank) for each document, then sums across lists. It works on ranks, not raw scores, which matters because BM25 scores and cosine similarities live on completely different scales, so averaging them is meaningless without careful normalisation. RRF sidesteps normalisation entirely and is robust to one retriever producing wildly larger numbers than the other.
Q: What is the difference between a bi-encoder and a cross-encoder, and where does each belong in a RAG pipeline?
A bi-encoder embeds the query and each document separately, so document vectors are precomputed and retrieval is a fast vector comparison, ideal for the first-stage wide search. A cross-encoder feeds the query and one document through the model together and outputs a relevance score, which is far more accurate but too slow to run over a whole corpus. So bi-encoder for recall in stage one, cross-encoder for precision reranking in stage two.
Q: When does keyword search beat semantic search?
When the query and the answer share exact, rare surface tokens: error codes, product SKUs, function names, specific numbers. Embeddings compress meaning and often blur these exact tokens together with similar-looking ones, so a semantic match can rank a distractor above the true answer. BM25 rewards the literal term match. That is the whole argument for hybrid search: keep both signals so neither blind spot sinks a query.
Q: How would you prevent an agentic RAG loop from running forever or leaking restricted data?
Two guards. For runaway loops, cap the number of retrieve-then-reason hops and add a completeness check so the loop exits as soon as it has the evidence it needs. For data leakage, filter by access control before ranking, never after, so an unauthorised chunk never enters the candidate set or the prompt. Both are cheap to add and both are the kind of thing a good reviewer will ask you to point to in the code.
Q: What is GraphRAG and what problem does it solve that vector RAG cannot?
GraphRAG stores knowledge as a graph of entities and relationships and answers by walking edges instead of matching passages. It solves multi-hop relational questions where the answer is a chain of facts that never co-occur in a single chunk, so similarity search cannot connect them. The traversal is straightforward graph code, the hard part is building a clean graph from messy documents, usually by having an LLM extract subject-relation-object triples.
Go deeper: the official Python documentation covers every edge case of this topic.
Related Posts
Previous: Python: Vector Databases (ChromaDB, Pinecone, pgvector for AI Apps)
Next: RAG Project: Build, Evaluate, and Ship a Retrieval App
Series Home: Python + AI/ML Tutorial Series
No comment