Frequently Asked Questions About Sujan Anand RAG Application Build Framework
22 answers covering everything from basics to advanced usage.
// Basics
What does the acronym RAG actually stand for?
RAG stands for Retrieval Augmented Generation. The three words describe the three-step process: Retrieval finds relevant chunks from your documents, Augmented adds those chunks to the prompt as context, and Generation produces the final grounded answer from the LLM. Understanding this order clarifies why retrieval quality determines answer quality.
What is an embedding in simple terms?
An embedding is a list of numbers — a vector — that represents the meaning of a piece of text. Similar meanings get similar numbers. In this framework, text-embedding-3-small produces 1536-dimensional vectors. Text goes in, a list of numbers comes out, and similar text gets similar numbers, which is what makes semantic search possible.
What is embedding space and why does it matter?
Embedding space is a giant map of all text where similar meanings live close together and unrelated meanings live far apart. 'Happy' and 'joyful' are neighbours; 'king' and 'pizza' are on opposite sides. Retrieval is just finding which stored chunks have coordinates closest to the question's coordinates — that proximity is the entire basis of semantic search.
What is the dot product and how is it used in RAG?
The dot product is a math operation that compares two embedding vectors: higher dot product means more similar meaning. It's computed as sum(x*y for x, y in zip(a, b)). During the validation phase you use it to confirm similar sentences score higher than unrelated ones. In production, ChromaDB uses vector similarity internally to rank retrieved chunks.
// How To
What tools and libraries do I need to install first?
Install openai, chromadb, python-dotenv, pymupdf (fitz), fastapi, uvicorn, python-multipart, and streamlit. Store your OpenAI API key in a .env file as OPENAI_API_KEY and load it with load_dotenv() and os.getenv() — never hardcode the key. These cover embedding, storage, PDF extraction, the backend server, and the chat frontend.
How do I validate embeddings before writing full RAG code?
Write a minimal script that embeds 2–3 sentences with text-embedding-3-small, extracts each vector via response.data[0].embedding, and computes dot product similarity between pairs. Confirm that semantically similar sentences score higher (above ~0.5) than unrelated ones (near 0.05). This proves the embedding space distinguishes relevant from irrelevant text before you invest in the full pipeline.
How do I implement chunk overlap correctly?
In your chunk_text function, use a while loop with start=0 and end=start+chunk_size, append text[start:end], then advance with start += chunk_size - chunk_overlap. This shifts the window forward by less than a full chunk, so consecutive chunks share ~50 characters like a Venn diagram. Skip chunks shorter than 50 characters to avoid noise.
How do I extract text from a PDF for RAG?
Use PyMuPDF (fitz): open the PDF with fitz.open(pdf_path), loop pages with enumerate(doc), call page.get_text().strip(), skip blank pages with 'if not text: continue', and store (page_number+1, text) tuples. Always call doc.close() when done to prevent memory leaks, especially when processing multiple PDFs.
How do I set up the FastAPI backend endpoints?
Create POST /ingest that accepts an UploadFile, saves it to a temp file, runs the Offline Pipeline, deletes the temp file, and returns chunk counts. Create POST /ask that accepts a Pydantic AskRequest(question, n_results=3), runs the Online Pipeline, and returns question, answer, and sources. Add a GET / health-check and enable CORS so the Streamlit frontend can reach it.
How many chunks should I retrieve per question?
The default is 3 (n_results=3), which balances context richness against token cost and noise. Retrieve more if your answers consistently miss information spread across multiple sections, but each extra chunk adds tokens and can dilute relevance. Inspect similarity scores — if the third chunk's score is already far from the question, adding more won't help.
// Troubleshooting
Why does my Streamlit chat history disappear on every message?
Because Streamlit reruns the whole script on every interaction. Store your chat history in st.session_state['messages'] so it persists across reruns. Without this, each new question wipes the conversation. Also use st.chat_input with the walrus operator (if question := st.chat_input(...)) to capture input cleanly.
Why does my RAG system return duplicate or conflicting answers after re-uploading a PDF?
You re-ingested without clearing the old data, so ChromaDB now holds duplicate chunks. Always delete the existing collection before re-ingesting the same document, then recreate it with get_or_create_collection. Duplicate chunk IDs also cause errors, so ensure each id is unique, like f'chunk_{chunk_index}'.
Why does my frontend show 'backend not reachable'?
The FastAPI server isn't running. Start it first in one terminal with 'uvicorn backend:app --reload', verify it at localhost:8000/docs, then launch the frontend in a second terminal with 'streamlit run frontend.py'. The Streamlit app POSTs to the backend, so the backend must be live before you upload or ask.
Why is my RAG system still hallucinating despite retrieving chunks?
Your prompt is likely missing the context-only instruction. Add 'Answer the question using only the context below. If the answer is not in the context, say I do not know.' Without this constraint, the LLM falls back on training data. Also check that retrieval is actually returning relevant chunks — poor chunking or wrong chunk size can starve the model of the right context.
Why are my embedding calls slow or hitting rate limits?
You're probably making one API call per chunk. Batch them — embed 100 chunks in a single call by looping with range(0, len(all_chunks), 100). Batched embedding is faster, cheaper, and production-ready, and it avoids the rate limits and latency of thousands of individual requests when indexing large documents.
// Comparisons
How does RAG compare to just increasing the LLM's context window?
Large context windows let you paste more text, but you still pay for every token every query, latency grows, and models lose focus on relevant details buried in huge inputs. RAG retrieves only the top few relevant chunks per question, keeping context small, cheap, and sharp. RAG also cites sources, which raw context stuffing cannot do reliably.
How does ChromaDB compare to Pinecone, Weaviate, or pgvector?
ChromaDB runs locally with zero account or setup via chromadb.PersistentClient(), making it ideal for learning and prototyping. Pinecone, Weaviate, and pgvector are production alternatives offering managed hosting, scale, and advanced features. This framework uses ChromaDB to teach the concepts; you can swap in a production vector database later with the same retrieval logic.
How does semantic search compare to keyword search for document Q&A?
Semantic search finds chunks by meaning proximity, so it answers 'how long do I have to return something?' from a document that says 'customers can return items within 30 days' — no shared keywords needed. Keyword search requires exact matches and fails on paraphrases, synonyms, and natural questions. For document Q&A, semantic search is dramatically more reliable.
// Advanced
How do I tune chunk size and n_results for my specific document type?
Start with the defaults — 500-character chunks, 50-character overlap, 3 results. For dense technical or legal text, smaller chunks (300 tokens) improve precision; for narrative or conversational content, larger chunks preserve context. Increase n_results if answers miss detail spread across sections, but watch token cost. Test with real questions and inspect the retrieved chunks and similarity scores to calibrate.
How do I add source citations and explainability to my RAG answers?
Store metadata — source filename, page number, and chunk index — with every chunk in ChromaDB via collection.add(metadatas=...). Include this metadata in your query results, return it alongside the answer, and instruct the LLM to mention the page. In the Streamlit UI, show retrieved chunks, page metadata, and similarity scores in an st.expander so users can verify every answer.
How do I move this RAG system from prototype to production?
Swap ChromaDB for a managed vector database like Pinecone or pgvector for scale and durability, add authentication and rate limiting to the FastAPI endpoints, tighten CORS from '*' to your real domains, add monitoring on retrieval quality and token spend, and implement incremental ingestion so you don't re-embed unchanged documents. Keep the two-pipeline architecture intact throughout.
Can I use a different embedding model or LLM with this framework?
Yes, but keep the embedding model consistent between ingestion and querying — you must embed questions with the same model used for chunks so they share coordinate space. You can swap gpt-4o-mini for another chat model in the generation step without re-embedding. Changing the embedding model, however, requires re-embedding your entire corpus.