Large Language Models (LLMs) make things up, and a Python RAG app is the cure. You saw the hallucination problem in the large language models tutorial: the model guesses when it does not actually know. RAG (Retrieval-Augmented Generation) stops that guessing by feeding it the right documents before it answers. Instead of leaning on whatever it memorized during training, you pull the relevant passages from your own files and paste them into the prompt.
The model then answers from real, checkable sources. Think of it like an open-book exam: a student who can look up the textbook gives far better answers than one working from a foggy memory. That is why RAG is the most important production pattern in Generative AI (GenAI). It solves the trust problem, because your users can click straight to the source.
In this post you build a complete RAG pipeline with LlamaIndex, the tool most teams reach for first when they need to search their own documents. You will load documents, chunk them into bite-sized pieces, turn those pieces into vectors, store the vectors, and then ask questions in plain English. By the end you will have a working system that answers questions about your own files and shows you which file each answer came from.
“The question of whether a computer can think is no more interesting than the question of whether a submarine can swim.”
Edsger Dijkstra
Last Updated: July 2026 | Tested on: Python 3.14.6, llama-index 0.14.22 | Difficulty: Advanced | Reading Time: 17 minutes
- large language models tutorial
- pip install llama-index chromadb
Retrieval-Augmented Generation (RAG) is the pattern of handing an LLM your own documents so it can answer questions about them accurately. The model’s training data may be stale, or it may have never seen your private files at all. RAG works around that. It fetches the relevant passages from your document collection and slips them into the prompt, so the model answers from your actual data instead of from memory.
RAG is the most practical way to build AI apps over private data without paying to fine-tune a model. This post walks through the full pipeline with LlamaIndex: loading documents, chunking, embedding, storing them in a vector index, and querying. By the end you will have a working RAG system that answers questions about your own documents and cites its sources.
Table of Contents
The RAG Pipeline
Every RAG system has two phases: ingestion (loading and indexing your documents) and querying (retrieving relevant chunks and generating answers). The diagram below shows the complete flow.
The diagram shows the RAG (Retrieval-Augmented Generation) pipeline end to end. On the ingestion side, documents are chunked and embedded into vectors that live in an index. On the query side, a user question is embedded and matched against that index to pull back the most relevant chunks, and those chunks are dropped into the LLM prompt as context before it writes an answer. This is how RAG sidesteps the model’s knowledge cutoff and its habit of making things up: every answer is grounded in your actual documents. Three knobs control both accuracy and cost, and they are the ones you will tune most: chunk size, the embedding model, and how many chunks you retrieve per question.
file_search tool and AWS Bedrock Knowledge Bases both index your files for you. One thing to avoid: do not build on OpenAI’s older Assistants API, which shuts down on August 26, 2026 (see the OpenAI deprecations page), use the Responses API instead. This space moves every few weeks, so always check each project’s own docs for the current version and API shape.Document Ingestion: Load, Chunk, Embed, Store
Ingestion is the one-time setup step. You run it whenever your documents change, not on every question. Picture a librarian taking a stack of new books, reading each one, writing index cards for the important bits, and filing those cards in a drawer. That filing is exactly what this script does. Here a developer named Rahul points LlamaIndex at a folder, it reads every file, cuts each file into chunks, turns each chunk into a vector (a list of numbers that captures its meaning), and saves the lot to disk so you never have to repeat the work. Every Python RAG pipeline, whatever the framework, starts with this filing step.
📄 rag_ingestion.py: building the knowledge base
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# Rahul builds a RAG system for his team's documentation
# Step 1: Configure the LLM and the embedding model.
# Model IDs churn every few weeks, so check the provider docs for the
# current names. At the time of writing (mid-2026) both of these are current.
Settings.llm = LlamaOpenAI(model="gpt-5.4-mini", temperature=0) # cheap, capable default tier
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
# Prefer not to depend on a paid, closed embedding model? Swap in a local
# open-weight one and nothing else in this pipeline changes:
# from llama_index.embeddings.huggingface import HuggingFaceEmbedding
# Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
# Step 2: Load documents from a directory
documents = SimpleDirectoryReader("./company_docs/").load_data()
print(f"Loaded {len(documents)} documents")
for doc in documents[:3]:
print(f" {doc.metadata.get('file_name', 'unknown')}: {len(doc.text)} chars")
# Step 3: Chunk documents intelligently
splitter = SentenceSplitter(
chunk_size=512, # ~512 tokens per chunk
chunk_overlap=50, # 50-token overlap for context continuity
)
nodes = splitter.get_nodes_from_documents(documents)
print(f"\nSplit into {len(nodes)} chunks")
print(f"Average chunk size: {sum(len(n.text) for n in nodes) // len(nodes)} chars")
# Step 4: Create vector index (embeds and stores)
index = VectorStoreIndex(nodes)
# Save for later use
index.storage_context.persist(persist_dir="./rag_index/")
print(f"\nIndex saved to ./rag_index/")
print(f"Ready for querying!")
▶ Output
Loaded 12 documents api_reference.md: 15234 chars deployment_guide.md: 8921 chars architecture_overview.md: 12456 chars Split into 87 chunks Average chunk size: 498 chars Index saved to ./rag_index/ Ready for querying!
SentenceSplitter chunking you see later in this post) we did run, and that output is real.What happened here: Four small steps and you have a searchable knowledge base. SimpleDirectoryReader read every file in the folder and handed back a list of documents. SentenceSplitter cut them into chunks of about 512 tokens, with a 50-token overlap so an idea that spans two chunks is not sliced clean in half. VectorStoreIndex then did the expensive part: it sent each chunk to the embedding model, got back a vector, and stored every chunk next to its vector. persist() wrote all of that to disk. Run this once, and every future question reuses the saved index instead of re-embedding the whole folder. That is the librarian’s drawer of index cards from earlier, now filled and filed.
Querying with Citations
With the index saved, asking a question is the easy part. Think of it like asking a librarian who has already filed every index card: you do not wait while the books get re-read, the answer comes straight from the drawer. Here a developer named Aditi loads the index back from disk and turns it into a query engine. When she asks a question, the engine embeds it the same way the chunks were embedded, finds the closest chunks by vector similarity, and feeds them to the model as context. This is the moment a Python RAG system earns its keep: the answer arrives with its sources attached.
The model writes the answer, and because we also keep the source chunks, we can show exactly which files the answer leaned on. That last part is what turns a black box into something you can trust.
📄 rag_query.py: answering questions with source citations
from llama_index.core import StorageContext, load_index_from_storage
# Aditi queries the RAG system
storage_context = StorageContext.from_defaults(persist_dir="./rag_index/")
index = load_index_from_storage(storage_context)
# Create query engine with citation support
query_engine = index.as_query_engine(
similarity_top_k=3, # Retrieve top 3 most relevant chunks
response_mode="compact", # Compact responses using all retrieved context
)
# Ask questions about company documentation
questions = [
"How do I deploy the application to production?",
"What authentication methods does the API support?",
"What is the recommended database setup for high availability?",
]
for q in questions:
response = query_engine.query(q)
print(f"\nQ: {q}")
print(f"A: {response.response[:300]}...")
print(f"\nSources:")
for node in response.source_nodes:
score = node.score if node.score else 0
source = node.node.metadata.get("file_name", "unknown")
print(f" [{score:.3f}] {source}: {node.node.text[:100]}...")
print("-" * 60)
▶ Output
Q: How do I deploy the application to production? A: To deploy to production, follow these steps: 1) Build the Docker image using `docker build -t myapp:latest .` 2) Push to your container registry with `docker push registry.example.com/myapp:latest` 3) Update the Kubernetes deployment manifest with the new image tag 4) Apply with `kubectl apply -f deploy.yaml`... Sources: [0.892] deployment_guide.md: ## Production Deployment\n\nThe recommended deployment... [0.834] architecture_overview.md: ### Infrastructure\n\nThe application runs on... [0.756] api_reference.md: ## Health Check Endpoint\n\nUse GET /health to verify... ------------------------------------------------------------ Q: What authentication methods does the API support? A: The API supports three authentication methods: 1) API key authentication via the X-API-Key header for server-to-server calls 2) JWT bearer tokens for user authentication via OAuth 2.0 3) Session-based auth for the web dashboard... Sources: [0.921] api_reference.md: ## Authentication\n\nAll API endpoints require authentication... [0.867] api_reference.md: ### OAuth 2.0 Flow\n\nFor user-facing applications... [0.712] deployment_guide.md: ## Security Configuration\n\nEnsure API keys are stored... ------------------------------------------------------------
What happened here: The query engine embedded each question, found the three closest chunks by vector similarity, dropped them into the LLM prompt as context, and asked the model to answer using only that context. Each source line carries a similarity score from 0 to 1: higher means a closer match. Because the prompt tells the model to stick to the retrieved chunks, it has far less room to invent things. It is not magic, retrieval can still miss, but a grounded answer with citations is a world away from a confident guess. And the source list means a reader can open the file and check for themselves.
Chunking Strategies: The Most Underrated RAG Decision
Here is the part most Python RAG builders get wrong. Your retrieval is only as good as your chunks. If a chunk is huge, it crams several unrelated ideas together, and the match score gets watered down. If a chunk is tiny, it loses the context that gave a sentence its meaning. Think of cutting a pizza: slice it too big and one piece has three different toppings fighting for attention, slice it too small and you cannot taste anything. You want pieces that are big enough to be satisfying on their own but small enough to be about one thing.
There are three common ways to cut. Here a developer named Vinay lines them up so you can see when each one earns its keep.
📄 chunking.py: how chunking strategy affects RAG quality
# Vinay compares three chunking strategies side by side
strategies = {
"Fixed-size (512 chars)": {
"method": "Split every 512 characters",
"pros": "Simple, predictable chunk size",
"cons": "Cuts mid-sentence, breaks context",
"best_for": "Quick prototyping",
},
"Sentence-based (512 tokens)": {
"method": "Split at sentence boundaries, max 512 tokens",
"pros": "Preserves sentence integrity, overlap handles boundaries",
"cons": "Unrelated sentences may share a chunk",
"best_for": "Most RAG applications (default choice)",
},
"Semantic chunking": {
"method": "Split where embedding similarity drops",
"pros": "Each chunk is semantically coherent",
"cons": "Slower (requires embedding computation), variable sizes",
"best_for": "High-quality RAG where accuracy matters most",
},
}
print("Chunking Strategy Comparison\n")
for name, info in strategies.items():
print(f" {name}")
print(f" Method: {info['method']}")
print(f" Pros: {info['pros']}")
print(f" Cons: {info['cons']}")
print(f" Best for: {info['best_for']}")
print()
▶ Output
Chunking Strategy Comparison
Fixed-size (512 chars)
Method: Split every 512 characters
Pros: Simple, predictable chunk size
Cons: Cuts mid-sentence, breaks context
Best for: Quick prototyping
Sentence-based (512 tokens)
Method: Split at sentence boundaries, max 512 tokens
Pros: Preserves sentence integrity, overlap handles boundaries
Cons: Unrelated sentences may share a chunk
Best for: Most RAG applications (default choice)
Semantic chunking
Method: Split where embedding similarity drops
Pros: Each chunk is semantically coherent
Cons: Slower (requires embedding computation), variable sizes
Best for: High-quality RAG where accuracy matters most
Tables are fine, but watching the splitter actually cut is better. The sentence-based splitter needs no Application Programming Interface (API) key, it only counts tokens and looks for sentence boundaries, so this next block really runs on your machine. Vinay feeds it a short paragraph and asks for small chunks so the cuts are easy to see.
📄 chunk_demo.py: a real sentence-based split you can run locally
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
# No API key needed: SentenceSplitter only counts tokens and
# cuts at sentence boundaries. Nothing is sent to a model.
text = (
"Retrieval-Augmented Generation grounds an LLM in your own documents. "
"Instead of trusting the model's memory, you fetch the relevant passages and "
"paste them into the prompt. The model then answers from real sources. "
"Chunking is the step that decides which passages can be fetched. "
"If a chunk is too big, it mixes unrelated ideas and dilutes the match. "
"If a chunk is too small, it loses the context around a single sentence. "
"A 512-token chunk with a small overlap is the sweet spot for most teams. "
"The overlap carries a few tokens from the previous chunk so a sentence that "
"straddles a boundary is never cut clean in half."
)
doc = Document(text=text)
splitter = SentenceSplitter(chunk_size=60, chunk_overlap=10)
nodes = splitter.get_nodes_from_documents([doc])
print(f"Original text: {len(text)} chars")
print(f"Split into {len(nodes)} chunks")
print()
for i, node in enumerate(nodes, start=1):
print(f"Chunk {i} ({len(node.text)} chars): {node.text}")
▶ Output
Original text: 620 chars Split into 3 chunks Chunk 1 (279 chars): Retrieval-Augmented Generation grounds an LLM in your own documents. Instead of trusting the model's memory, you fetch the relevant passages and paste them into the prompt. The model then answers from real sources. Chunking is the step that decides which passages can be fetched. Chunk 2 (215 chars): If a chunk is too big, it mixes unrelated ideas and dilutes the match. If a chunk is too small, it loses the context around a single sentence. A 512-token chunk with a small overlap is the sweet spot for most teams. Chunk 3 (124 chars): The overlap carries a few tokens from the previous chunk so a sentence that straddles a boundary is never cut clean in half.
What happened here: The splitter never cut a sentence in the middle. It packed whole sentences into each chunk until it hit the 60-token budget, then started a fresh chunk. That is the whole point of sentence-based splitting: every chunk is a clean unit of meaning. In a real pipeline you would use chunk_size=512, not 60, but the behavior is identical, just with bigger pieces. This is the same SentenceSplitter the ingestion script used earlier, so what you see here is exactly how your documents get cut before they are embedded.
Common Mistakes
- Chunks too large: 2,000-token chunks dilute relevance. Smaller chunks (256-512 tokens) with overlap produce better retrieval because each chunk is more focused.
- No chunk overlap: Without overlap, context at chunk boundaries is lost. Use 10-15% overlap (e.g., 50-token overlap for 512-token chunks).
- Not evaluating retrieval quality: If the retriever returns irrelevant chunks, the best LLM in the world cannot generate good answers. Always check what is actually being retrieved before blaming the LLM.
Practice Exercises
- Exercise 1, your first RAG: Drop three or four of your own notes or README files into a folder, run the ingestion script, then ask the query engine two questions about them. Confirm the answers come from your files and not from the model’s memory.
- Exercise 2, tune the chunks: Take the local
chunk_demo.pyand run it three times withchunk_sizeset to 30, 60, and 120. Watch how the number of chunks and the cut points change. Which size keeps each idea together best? - Exercise 3, inspect retrieval: Before trusting any answer, print
response.source_nodesfor a question you know the answer to. If the top chunk is the wrong one, that is your bug, not the model. Fix retrieval first.
More in this series:
- LLM Observability with Langfuse: Traces, Spans, and Cost
- RAG Project: Build, Evaluate, and Ship a Retrieval App
- GenAI: Fine-Tuning LLMs with LoRA, QLoRA, and PEFT
Frequently Asked Questions
When should I use RAG vs fine-tuning?
Use RAG when you need access to specific, frequently changing documents (company docs, knowledge bases, product catalogs). Use fine-tuning (fine-tuning tutorial) when you need the model to learn a specific behavior or style. Many production Python RAG apps use both: fine-tuning for style and retrieval for knowledge.
What types of documents can I use with RAG?
Through LlamaHub, LlamaIndex reads a huge range of sources: PDF, Word, PowerPoint, HTML, Markdown, CSV, JSON, databases (SQL, MongoDB), and APIs (Notion, Slack, Google Docs), among many others. Use SimpleDirectoryReader for plain files and the specialized readers for APIs and databases. Check the LlamaHub docs for the current loader list.
How accurate is RAG compared to asking the LLM directly?
For factual questions about your own documents, RAG sharply reduces hallucination versus asking the model cold, because the answer is grounded in retrieved text instead of memory. The whole thing hinges on retrieval quality. If the right chunks are retrieved, the model almost always gets the answer right. If the wrong chunks are retrieved, RAG can actually do worse than a plain query, since the model is now anchored to irrelevant context. That is why you always check what is being retrieved before you trust the answers.
Interview Questions on Python RAG
The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.
Q: What are the two phases of a RAG pipeline, and what happens in each?
Ingestion and querying. Ingestion is a one-time step that loads documents, splits them into chunks, embeds each chunk into a vector, and stores those vectors in an index. Querying embeds the user’s question, retrieves the most similar chunks, and hands them to the LLM as context so it answers from your data instead of its memory.
Q: Why add overlap between chunks?
Overlap carries a few tokens from the end of one chunk into the start of the next, so an idea or sentence that straddles a boundary is not cut clean in half. Without it, retrieval can miss context that sits right at a chunk edge. A common setting is 10 to 15 percent, for example a 50-token overlap on a 512-token chunk.
Q: Your RAG app returns confident answers that are flat wrong, even though the documents contain the correct information. What do you check first?
Check retrieval before blaming the LLM. Print response.source_nodes and look at which chunks came back and their similarity scores. If the right chunk is not in the top-K, the problem is retrieval, so revisit chunk size, overlap, the embedding model, or raise similarity_top_k. The model can only answer from the context it was handed, so a bad retrieval guarantees a bad answer.
Q: Ingestion is slow and your API bill spikes every time you restart the app. What is wrong?
You are most likely re-embedding the whole corpus on every startup instead of persisting and reloading the index. Embedding is the expensive API call. Run ingestion once, call index.storage_context.persist(), and on later runs use load_index_from_storage to read the saved index from disk so you never pay to re-embed unchanged documents.
Q: What does similarity_top_k control, and what is the trade-off?
It sets how many of the most similar chunks the retriever pulls into the prompt as context. Set it too low and you may miss the chunk that holds the answer; set it too high and you pad the prompt with noise, raise token cost, and can distract the model. Three to five is a common starting point, then tune it based on what actually gets retrieved.
Q: What is an embedding, and why does RAG rely on it?
An embedding is a list of numbers (a vector) that captures the meaning of a piece of text, so texts with similar meaning sit close together in vector space. RAG embeds both your chunks and the user’s question, then finds the chunks whose vectors are nearest the question’s vector. That similarity search is what lets it fetch relevant passages even when the wording does not match exactly.
What’s Next?
You now have a working Python RAG pipeline: you loaded documents, chunked them with SentenceSplitter, embedded and stored them in a vector index, and queried that index to get answers that cite their sources. The biggest lever, as you saw, is chunking, so tune chunk size and overlap before you reach for a fancier model. Next, in the vector databases tutorial, we go deeper into the storage layer, comparing ChromaDB, Pinecone, and pgvector so you can pick the right vector database for your scale and budget.
Want the full path from Python basics to production AI? Explore the complete Python + AI/ML tutorial series home.
Related Posts
Previous: LLM Observability with Langfuse: Traces, Spans, and Cost
Next: Python: Vector Databases (ChromaDB, Pinecone, pgvector for AI Apps)
Series Home: Python + AI/ML Tutorial Series
No comment