An LLM's knowledge is frozen at training time (fundamentals Module 15). Ask it about your company's internal docs, a PDF you just uploaded, or today's news — it either says "I don't know" or, worse, hallucinates a confident wrong answer.
RAG solves this without retraining: at question time, retrieve the relevant facts from your own data (using the vector DB from Module 10) and hand them to the model as context. Now it answers from real sources.
RAG is just Retrieve → Augment → Generate. Everything you've learned plugs in. Click each step to see its role.
You already have every piece: a vector DB (Module 10) and an LLM. RAG just wires them together.
# 1. RETRIEVE — find relevant chunks from your vector DB (Module 10)
results = collection.query(query_texts=[question], n_results=3)
context = "\n".join(results["documents"][0])
# 2. AUGMENT — build a prompt that includes the retrieved context
prompt = f"""Answer using ONLY this context:
{context}
Question: {question}"""
# 3. GENERATE — let the LLM answer from the real facts
answer = llm.generate(prompt)Notice "Answer using ONLY this context". This tells the model to ground its answer in the retrieved facts and not fall back on its (possibly wrong) memory. Good RAG prompts also say "if the answer isn't in the context, say you don't know" — which slashes hallucinations.
Ask a question about a small "knowledge base." Watch RAG retrieve the right chunks, build the augmented prompt, and generate a grounded answer. (Simulated to show the flow; the code is above.)
You can't embed a whole 100-page PDF as one vector — it'd be too blurry to match anything. So you chunk it: split into smaller pieces, embed each. But how you chunk decides whether retrieval works. Chunks too big = imprecise; too small = lose context; split mid-sentence = broken meaning.
Good chunking uses overlap — each chunk shares a bit with its neighbors — so a fact that sits on a boundary isn't cut in half. A common recipe: ~500 tokens per chunk with ~50 tokens of overlap. Chunking quality is often the difference between RAG that works and RAG that doesn't.
Slide chunk size and overlap and watch how the document splits. See why too-small loses context and overlap prevents boundary cuts.
RAG is only as good as what it retrieves. If retrieval pulls the wrong chunks, the LLM gets bad context and gives a bad answer — "garbage in, garbage out." For each question, pick which retrieved chunk would actually help answer it.
A crucial real decision. RAG adds knowledge (facts the model can look up). Fine-tuning (Module 7) changes behavior/style. They solve different problems — and are often combined. Match each need.
Two phases: indexing (done once, ahead of time) and querying (every question). Put all the steps in the right order.
Same steps as Game 6, but now animated. First the system prepares your documents once. Later, every user question searches those prepared chunks and builds a grounded prompt.
This is the real version of the game: chunk three policy documents, embed the chunks, retrieve the closest chunks for a question, then use a QA model to extract the final answer from the retrieved context.
!pip -q install sentence-transformers transformers accelerate
import torch
from sentence_transformers import SentenceTransformer, util
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
# -----------------------------
# 1. Your tiny document database
# -----------------------------
documents = [
{
"title": "Refund Policy",
"text": """
Customers can request a refund within 14 days of purchase.
A receipt or order ID is required for all refund requests.
Digital products are not refundable after they have been downloaded.
Refunds usually take 5 to 7 business days to appear on the card.
"""
},
{
"title": "Shipping Policy",
"text": """
Standard shipping takes 3 to 5 business days.
Express shipping takes 1 to 2 business days and costs extra.
International shipping may take 7 to 14 business days.
Customers receive a tracking link by email after the order ships.
"""
},
{
"title": "Support Policy",
"text": """
Customer support is available Monday through Friday from 9 AM to 6 PM EST.
Weekend support is available by email only.
Urgent account security issues are prioritized within 24 hours.
"""
}
]
# -----------------------------
# 2. Chunk the documents
# The overlap keeps boundary facts from being cut in half.
# -----------------------------
def chunk_text(text, chunk_size=35, overlap=8):
words = text.split()
chunks = []
step = chunk_size - overlap
for start in range(0, len(words), step):
end = start + chunk_size
chunk = " ".join(words[start:end])
chunks.append(chunk)
if end >= len(words):
break
return chunks
all_chunks = []
for doc in documents:
chunks = chunk_text(doc["text"], chunk_size=35, overlap=8)
for i, chunk in enumerate(chunks):
all_chunks.append({
"title": doc["title"],
"chunk_id": i,
"text": chunk
})
print(f"Total chunks created: {len(all_chunks)}")
for chunk in all_chunks:
print(f"\n[{chunk['title']} | chunk {chunk['chunk_id']}]")
print(chunk["text"])
# -----------------------------
# 3. Embed all chunks
# Each chunk becomes a 384-number meaning vector.
# -----------------------------
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunk_texts = [chunk["text"] for chunk in all_chunks]
chunk_embeddings = embedding_model.encode(
chunk_texts,
convert_to_tensor=True,
normalize_embeddings=True
)
print("\nEmbedding shape:", chunk_embeddings.shape)
# -----------------------------
# 4. Retrieve relevant chunks
# Embed the question, then find chunks with highest cosine similarity.
# -----------------------------
def retrieve(question, top_k=2):
question_embedding = embedding_model.encode(
question,
convert_to_tensor=True,
normalize_embeddings=True
)
scores = util.cos_sim(question_embedding, chunk_embeddings)[0]
top_results = torch.topk(scores, k=top_k)
retrieved = []
for score, index in zip(top_results.values, top_results.indices):
chunk = all_chunks[index]
retrieved.append({
"score": score.item(),
"title": chunk["title"],
"chunk_id": chunk["chunk_id"],
"text": chunk["text"]
})
return retrieved
# -----------------------------
# 5. Load a QA model
# It reads the retrieved context and extracts the answer span.
# -----------------------------
qa_model_name = "deepset/minilm-uncased-squad2"
qa_tokenizer = AutoTokenizer.from_pretrained(qa_model_name)
qa_model = AutoModelForQuestionAnswering.from_pretrained(qa_model_name)
def answer_from_context(question, context):
inputs = qa_tokenizer(
question,
context,
return_tensors="pt",
truncation=True,
max_length=512
)
with torch.no_grad():
outputs = qa_model(**inputs)
start_idx = torch.argmax(outputs.start_logits)
end_idx = torch.argmax(outputs.end_logits)
if end_idx < start_idx:
return "I could not find the answer in the retrieved context."
answer_ids = inputs["input_ids"][0][start_idx:end_idx + 1]
answer = qa_tokenizer.decode(answer_ids, skip_special_tokens=True)
if answer.strip() == "":
return "I could not find the answer in the retrieved context."
return answer
# -----------------------------
# 6. Full RAG function
# Retrieve first, augment the prompt/context, then answer.
# -----------------------------
def rag_answer(question):
retrieved_chunks = retrieve(question, top_k=2)
context = "\n".join([chunk["text"] for chunk in retrieved_chunks])
answer = answer_from_context(question, context)
print("\n" + "=" * 70)
print("QUESTION:")
print(question)
print("\nRETRIEVED CHUNKS:")
for chunk in retrieved_chunks:
print(f"\n- {chunk['title']} | chunk {chunk['chunk_id']} | score: {chunk['score']:.3f}")
print(chunk["text"])
print("\nAUGMENTED PROMPT GIVEN TO QA MODEL:")
print("Use this retrieved context to answer the question:")
print(context)
print("Question:", question)
print("\nFINAL ANSWER:")
print(answer)
# -----------------------------
# 7. Try questions
# -----------------------------
rag_answer("How many days do I have to request a refund?")
rag_answer("How long does express shipping take?")
rag_answer("When is customer support available?")The embedding model is the retriever. It found the right policy chunks with cosine similarity scores like 0.843 and 0.792. The QA model did not search the documents by itself; it only read the retrieved context and extracted the final answer.
Click each question to see what RAG retrieved and what answer came out. This is the same output from the notebook run, cleaned up into a visual reader.
Total chunks created: 5
Embedding shape: torch.Size([5, 384])
QUESTION:
How many days do I have to request a refund?
RETRIEVED CHUNKS:
- Refund Policy | chunk 0 | score: 0.718
- Refund Policy | chunk 1 | score: 0.569
FINAL ANSWER:
14
QUESTION:
How long does express shipping take?
RETRIEVED CHUNKS:
- Shipping Policy | chunk 0 | score: 0.843
- Shipping Policy | chunk 1 | score: 0.566
FINAL ANSWER:
1 to 2 business days
QUESTION:
When is customer support available?
RETRIEVED CHUNKS:
- Support Policy | chunk 0 | score: 0.792
- Shipping Policy | chunk 1 | score: 0.425
FINAL ANSWER:
monday through fridayThis version is closer to a small production RAG system: the learner uploads one real PDF, the code reads the whole document, chunks it, stores vectors in ChromaDB, retrieves relevant chunks, applies an abstain threshold, and asks a local LLM to answer only from the retrieved text.
!pip -q install chromadb sentence-transformers transformers accelerate pypdf ftfy
import re
import torch
import chromadb
from google.colab import files
from pypdf import PdfReader
from ftfy import fix_text
from sentence_transformers import SentenceTransformer
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
# 1. Upload one PDF.
uploaded = files.upload()
if len(uploaded) != 1:
raise ValueError("Upload exactly one PDF for this test.")
file_name = list(uploaded.keys())[0]
print("Uploaded:", file_name)
# 2. Read the whole PDF.
def read_pdf(path):
reader = PdfReader(path)
pages = []
for page_number, page in enumerate(reader.pages, start=1):
text = page.extract_text() or ""
pages.append({"page": page_number, "text": text})
return pages
pages = read_pdf(file_name)
raw_text = "\n".join(
f"[Page {p['page']}]\n{p['text']}"
for p in pages
)
raw_text = fix_text(raw_text)
raw_text = re.sub(r"\s+", " ", raw_text).strip()
print("Pages read:", len(pages))
print("Characters read:", len(raw_text))
print("\nPreview:")
print(raw_text[:1000])
# 3. Chunk the document.
def chunk_text(text, chunk_size=120, overlap=30):
words = text.split()
if not words:
return []
chunks = []
step = chunk_size - overlap
for start in range(0, len(words), step):
end = start + chunk_size
chunk = " ".join(words[start:end])
page_matches = re.findall(r"\[Page (\d+)\]", chunk)
page = int(page_matches[-1]) if page_matches else None
chunks.append({
"chunk_id": len(chunks),
"start_word": start,
"end_word": min(end, len(words)),
"page": page,
"text": chunk
})
if end >= len(words):
break
return chunks
chunks = chunk_text(raw_text, chunk_size=120, overlap=30)
if len(chunks) == 0:
raise ValueError("No chunks created. If this PDF is scanned, OCR is needed.")
print("\nTotal chunks:", len(chunks))
for chunk in chunks:
print(f"\n--- chunk {chunk['chunk_id']} | page {chunk['page']} | words {chunk['start_word']} to {chunk['end_word']} ---")
print(chunk["text"][:500])
# 4. Embed chunks.
embedding_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunk_texts = [chunk["text"] for chunk in chunks]
chunk_embeddings = embedding_model.encode(
chunk_texts,
normalize_embeddings=True
).tolist()
print("\nEmbedding count:", len(chunk_embeddings))
print("Embedding dimensions:", len(chunk_embeddings[0]))
# 5. Store vectors in ChromaDB.
client = chromadb.PersistentClient(path="./refund_policy_chroma_db")
collection_name = "refund_policy_rag"
try:
client.delete_collection(collection_name)
except Exception:
pass
collection = client.get_or_create_collection(
name=collection_name,
metadata={"hnsw:space": "cosine"}
)
collection.add(
ids=[f"{file_name}_chunk_{chunk['chunk_id']}" for chunk in chunks],
documents=[chunk["text"] for chunk in chunks],
embeddings=chunk_embeddings,
metadatas=[
{
"file_name": file_name,
"chunk_id": chunk["chunk_id"],
"page": chunk["page"] if chunk["page"] is not None else -1,
"start_word": chunk["start_word"],
"end_word": chunk["end_word"]
}
for chunk in chunks
]
)
print("\nStored in ChromaDB collection:", collection_name)
print("Vector DB count:", collection.count())
# 6. Load a small local instruction model.
llm_name = "google/flan-t5-base"
tokenizer = AutoTokenizer.from_pretrained(llm_name)
llm = AutoModelForSeq2SeqLM.from_pretrained(llm_name)
device = "cuda" if torch.cuda.is_available() else "cpu"
llm = llm.to(device)
print("\nLLM loaded:", llm_name)
print("Device:", device)
# 7. Retrieve relevant chunks.
def retrieve(question, top_k=4):
question_embedding = embedding_model.encode(
question,
normalize_embeddings=True
).tolist()
results = collection.query(
query_embeddings=[question_embedding],
n_results=min(top_k, len(chunks)),
include=["documents", "metadatas", "distances"]
)
retrieved = []
for doc, meta, distance in zip(
results["documents"][0],
results["metadatas"][0],
results["distances"][0]
):
retrieved.append({
"text": doc,
"metadata": meta,
"distance": distance,
"similarity": 1 - distance
})
return retrieved
# 8. Build cited context for the LLM.
def build_context(retrieved):
blocks = []
for item in retrieved:
meta = item["metadata"]
blocks.append(
f"[Source: chunk {meta['chunk_id']}, page {meta['page']}, similarity {item['similarity']:.3f}]\n"
f"{item['text']}"
)
return "\n\n".join(blocks)
# 9. Ask using RAG.
def answer_with_rag(question, top_k=4, min_similarity=0.38):
retrieved = retrieve(question, top_k=top_k)
best_similarity = retrieved[0]["similarity"] if retrieved else 0
print("\n" + "=" * 80)
print("QUESTION:")
print(question)
print("\nBEST SIMILARITY:", round(best_similarity, 3))
print("\nRETRIEVED CHUNKS:")
for item in retrieved:
meta = item["metadata"]
print(f"\n- chunk {meta['chunk_id']} | page {meta['page']} | similarity: {item['similarity']:.3f}")
print(item["text"][:800])
if best_similarity < min_similarity:
final_answer = "I don't know based on the uploaded document."
print("\nFINAL ANSWER:")
print(final_answer)
return {"question": question, "answer": final_answer, "retrieved": retrieved}
context = build_context(retrieved)
prompt = f"""
You are a careful refund-policy question-answering assistant.
Use ONLY the provided context.
Do not use outside knowledge.
If the answer is not directly stated in the context, say:
"I don't know based on the uploaded document."
Give a direct answer.
If there are multiple conditions, list them as bullet points.
Include the source chunk and page in parentheses.
CONTEXT:
{context}
QUESTION:
{question}
ANSWER:
""".strip()
inputs = tokenizer(
prompt,
return_tensors="pt",
truncation=True,
max_length=1024
).to(device)
with torch.no_grad():
output_ids = llm.generate(
**inputs,
max_new_tokens=220,
do_sample=False,
num_beams=4
)
llm_answer = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print("\nLLM ANSWER:")
print(llm_answer)
return {
"question": question,
"answer": llm_answer,
"retrieved": retrieved,
"best_similarity": best_similarity
}
# 10. Test questions.
answer_with_rag("How long does WANT TS take to pay refunds after application?", top_k=4)
answer_with_rag("When is there no refund applicable?", top_k=4)
answer_with_rag("How does a student request a refund?", top_k=4)
answer_with_rag("What is the fee for Enrolment cancellation fee?", top_k=4)
answer_with_rag("What is the company's hiring policy?", top_k=4)Retrieval and generation are separate quality checks. In the refund-policy run, vector retrieval found the correct chunks with strong similarities, but the small local LLM sometimes copied too much context or chose the wrong neighboring row. Real production RAG usually adds reranking, stricter answer extraction, stronger models, and citations.
This run used a real uploaded PDF: Refund-Policy-v2023.pdf. The index was small enough to inspect manually, which makes it perfect for learning how to judge RAG accuracy.
Next up → Module 12: Memory in AI (the finale!) — RAG retrieves from documents, but a chatbot also needs to remember your conversation. We'll cover the types of memory (short-term/context, long-term/vector, episodic, working) and exactly when to use each.