Sujan Anand RAG Application Build Framework

// When should you use the RAG Application Build Framework?

Use this skill whenever you need to build an AI system that must answer questions from a specific document corpus, knowledge base, or PDF collection — especially when accuracy, source grounding, and no hallucinations are non-negotiable.

// What do you need before building a RAG system?

• Document corpus or PDFrequired
  The source document(s) the RAG system will read and answer from — e.g. a PDF, knowledge base, or set of text files
• OpenAI API keyrequired
  A valid OpenAI API key with credit loaded, obtained from platform.openai.com, used for embedding and generation calls
• User question(s)required
  The natural language questions end-users will ask against the ingested documents
• Chunk size and overlap settings
  Characters per chunk (default 500) and overlap between consecutive chunks (default 50); tune based on document type
• Number of results (n_results)
  How many top chunks to retrieve per query (default 3); controls how much context the LLM receives

// What are the core principles behind a reliable RAG system?

// How do you build a RAG system step by step?

1. 1

   Install dependencies

   Install: openai, chromadb, python-dotenv, pymupdf (fitz), fastapi, uvicorn, python-multipart, streamlit. Store your OpenAI API key in a .env file as OPENAI_API_KEY. Never hardcode the key. Load it with load_dotenv() and os.getenv('OPENAI_API_KEY').

2. 2

   Validate your understanding of embeddings before writing RAG code

   Write a minimal script: embed 2–3 sentences using client.embeddings.create(input=sentence, model='text-embedding-3-small'). Extract the vector via response.data[0].embedding. Compute dot product similarity between pairs. Confirm that semantically similar sentences score higher than unrelated ones. Each embedding is 1536 numbers — these are not random; together they encode meaning as mathematical coordinates.

3. 3

   Build the Offline Pipeline — extract text from documents

   Use PyMuPDF (fitz): open the PDF with fitz.open(pdf_path), loop through pages with enumerate(doc), call page.get_text().strip() to extract text, skip blank pages (if not text: continue), store tuples of (page_number+1, text) in a pages list, then doc.close() to prevent memory leaks.

4. 4

   Build the Offline Pipeline — chunk the extracted text

   Implement chunk_text(text, chunk_size=500, chunk_overlap=50). Use a while loop: start=0, end=start+chunk_size, append text[start:end], then advance with start += chunk_size - chunk_overlap. Skip tiny chunks shorter than 50 characters. This creates overlapping windows so sentences at boundaries are never lost.

5. 5

   Build the Offline Pipeline — embed chunks in batches

   Implement embed_texts(texts): call client.embeddings.create(input=texts, model='text-embedding-3-small') and return [item.embedding for item in response.data]. Process chunks in batches of 100 (loop with range(0, len(all_chunks), 100)) to avoid API overload and reduce cost.

6. 6

   Build the Offline Pipeline — store embeddings in ChromaDB with metadata

   Use chromadb.PersistentClient(path='./chroma_db') for persistence. Call get_or_create_collection('rag_collection'). Store with collection.add(documents=all_chunks, embeddings=all_embeddings, metadatas=all_metadatas, ids=all_ids). Each metadata dict must include source filename and page number. Each id must be unique (e.g. f'chunk_{chunk_index}'). If re-ingesting, delete the existing collection first to avoid duplicates.

7. 7

   Build the Online Pipeline — embed the user's question

   Use the same model (text-embedding-3-small) to embed the user question: embed_texts([question])[0]. This converts the question into the same coordinate space as your stored chunks, making semantic comparison valid.

8. 8

   Build the Online Pipeline — perform vector search (semantic search)

   Call collection.query(query_embeddings=[question_embedding], n_results=3, include=['documents','metadatas','distances']). ChromaDB compares the question vector against all stored chunk vectors by coordinate proximity — not keyword matching. Extract: chunks=results['documents'][0], metadatas=results['metadatas'][0], distances=results['distances'][0]. Higher dot-product similarity = more relevant chunk.

9. 9

   Build the Online Pipeline — inject retrieved chunks into a grounded prompt

   Build context = '\n\n'.join(retrieved_chunks). Construct prompt: 'You are a helpful assistant. Answer the question using only the context below. Always mention the page the answer came from. If the answer is not in the context, say I do not know.\n\nContext:\n{context}\n\nQuestion: {question}'. This constraint is what prevents hallucinations.

10. 10

    Build the Online Pipeline — call the LLM and return the grounded answer

    Call client.chat.completions.create(model='gpt-4o-mini', messages=[{'role':'user','content':prompt}]). Extract answer = response.choices[0].message.content. Return both the answer and the sources list (chunk text + metadata + distance score) for explainability.

11. 11

    Wrap the backend in a FastAPI server with two endpoints

    POST /ingest: accepts an UploadFile, saves to a temp file (tempfile.NamedTemporaryFile, delete=False), runs the full Offline Pipeline, deletes the temp file with os.unlink(), returns {message, chunks_added, total_chunks}. POST /ask: accepts AskRequest(question: str, n_results: int = 3) as a Pydantic BaseModel, runs the full Online Pipeline, returns {question, answer, sources}. Add a GET / health-check endpoint returning {status:'running', total_chunks}. Enable CORS middleware (allow_origins=['*']) so the frontend can reach the backend.

12. 12

    Build the Streamlit frontend with chat UI and source display

    Use st.set_page_config(layout='wide'). Use st.sidebar for PDF upload (st.file_uploader, type=['pdf']) and an 'Ingest PDF' button that POSTs to /ingest with a spinner. Store chat history in st.session_state['messages'] (without this, chat disappears on every rerun). Use st.chat_input for questions (walrus operator: if question := st.chat_input(...)). For each answer, display sources in an st.expander showing chunk text, page metadata, and similarity score. Run with: streamlit run frontend.py

13. 13

    Run and test the full stack

    Terminal 1: uvicorn backend:app --reload (starts FastAPI on localhost:8000). Verify at localhost:8000/docs — FastAPI auto-generates interactive API docs for both endpoints. Terminal 2: streamlit run frontend.py. Upload a PDF, click Ingest PDF, watch chunks count appear, then ask questions in the chat box. Verify that answers cite page sources and that unrelated questions return 'I do not know' rather than hallucinated answers.

// What are real examples of this RAG framework in action?

// What mistakes should you avoid when building RAG?

• Dumping an entire PDF as one embedding — embedding models have token limits and you cannot retrieve relevant sub-sections from a single monolithic embedding; always chunk first.
• Choosing chunk sizes that are too small (single sentences lose context) or too large (entire chapters add noise and inflate token costs); stay in the 300–500 token sweet spot.
• Forgetting chunk overlap — if you split without overlap, sentences at chunk boundaries are split in half and neither chunk makes sense; always set overlap to ~50 characters.
• Using a regular (exact-match) database instead of a vector database — keyword search cannot find 'return items within 30 days' when the question is 'how long do I have to return something'; you need semantic similarity search.
• Not including the context-only instruction in your prompt — without 'Answer only using the context below; if not in context say I do not know', the LLM will hallucinate from its training data.
• Making one API call per chunk when embedding large documents — always batch 100 chunks per API call to avoid rate limits, reduce latency, and lower cost.
• Forgetting to store metadata (source filename and page number) with each chunk — without it you cannot show users where answers came from, making the system untrustworthy in production.
• Not calling doc.close() after extracting text with PyMuPDF — this causes memory leaks when processing multiple PDFs.
• Re-ingesting a PDF without deleting the existing ChromaDB collection first — duplicate chunk IDs will cause errors; always delete then recreate the collection on re-ingest.
• Running the frontend without the backend server active — the Streamlit app will show 'backend not reachable' errors; always start uvicorn first, then streamlit.

// What are the key RAG terms you need to know?

RAG (Retrieval Augmented Generation)

A technique where, instead of asking an LLM to remember everything, you give it a search engine over your own data right before it answers. The three words — Retrieval, Augmented, Generation — describe the three-step process: retrieve relevant chunks, augment the prompt with them, generate a grounded answer.

Embedding

A list of numbers (a vector) that represents the meaning of a piece of text. Similar meanings get similar numbers. In this framework, text-embedding-3-small produces 1536-dimensional vectors. Text goes in, a list of numbers comes out; similar text gets similar numbers.

Embedding Space

A giant map of all text where similar meanings live close together (e.g. 'happy' and 'joyful' are neighbours; 'king' and 'pizza' are on opposite sides). Each piece of text has coordinates on this map — those coordinates are its embedding.

Vector Database

A database that stores embeddings and finds the most similar ones fast — searching by meaning proximity rather than exact keyword match. Described as 'a librarian who has read every book and finds the ones closest to your vibe in 2 seconds'.

ChromaDB

The local vector database used in this framework. Runs on your machine, requires zero account or setup, and is used via chromadb.PersistentClient(). Suitable for learning and prototyping; production alternatives include Pinecone, Weaviate, and pgvector.

Chunking

Splitting a document into small, overlapping pieces before embedding. Required because embedding models have token limits and retrieving an entire 100-page PDF as context confuses the LLM and wastes tokens. Each small piece is called a chunk.

Chunk Overlap

The number of characters that consecutive chunks share. Like a Venn diagram between adjacent chunks, overlap ensures sentences at chunk boundaries are never lost. Implemented as: start += chunk_size - chunk_overlap.

Offline Pipeline

The one-time ingestion process: extract text from document → split into chunks → embed each chunk → store embeddings and metadata in ChromaDB. Run once per document; does not run per query.

Online Pipeline

The per-query retrieval and generation process: embed the user's question → search ChromaDB for the nearest chunks → inject those chunks into a context-only prompt → LLM generates the grounded answer. Runs every time a user asks a question.

Semantic Search

Finding relevant chunks by vector proximity (meaning similarity) rather than exact keyword matching. The reason RAG can answer 'how long do I have to return something?' from a document that says 'customers can return items within 30 days' — no exact keyword overlap required.

Context-Only Prompt

The prompt instruction that constrains the LLM to only use retrieved chunks: 'Answer the question using only the context below. If the answer is not in the context, say I do not know.' This single instruction is what prevents hallucinations in a RAG system.

Batched Embedding

Processing 100 chunks in a single API call instead of one call per chunk. Faster, cheaper, and production-ready. Implemented by iterating over chunks with range(0, len(all_chunks), batch_size) where batch_size=100.

Dot Product

The mathematical operation used to compare two embedding vectors. Higher dot product = more similar meaning. Computed as sum(x*y for x, y in zip(a, b)). Used to validate that similar sentences score higher than unrelated ones during the embedding understanding phase.

Metadata

Information stored alongside each chunk in ChromaDB, including source filename and page number. Enables source tracking, citations, and RAG explainability — showing users exactly which document and page each answer came from.

Ingest

The act of running the full Offline Pipeline on a document: extract → chunk → embed → store. Exposed as the POST /ingest endpoint in the FastAPI backend.

// FREQUENTLY ASKED QUESTIONS