How to Learn RAG End-to-End as a Data Scientist
For Data scientists and ML engineers new to RAG · Based on Sujan Anand RAG Application Build Framework
// TL;DR
This guide gives data scientists and ML engineers a first-principles path to learning RAG using the RAG Application Build Framework. Start by validating embeddings with a dot-product script to see how meaning becomes coordinates, then build the Offline Pipeline (chunk, embed, store) and Online Pipeline (embed question, vector search, inject context, generate) as two clearly separated systems. Use ChromaDB locally with zero setup. By the end you understand embedding space, semantic search, chunking trade-offs, and hallucination control — the transferable fundamentals behind every production RAG system.
Why should you validate embeddings before writing any RAG code?
Because RAG's entire behavior depends on embeddings encoding meaning as coordinates, and you should prove that to yourself first. Write a minimal script: embed two or three sentences with text-embedding-3-small, pull each vector via response.data[0].embedding, and compute the dot product between pairs. You'll see semantically similar sentences score above ~0.5 while unrelated ones sit near 0.05. Each embedding is 1536 numbers that together encode meaning — not random noise. Internalizing this makes every later step, from retrieval to chunking, intuitive rather than mysterious.
How do the two pipelines separate concerns?
The framework's most important mental model is the Two Pipelines Architecture. The Offline Pipeline runs once per document: extract text with PyMuPDF, chunk it into overlapping windows, embed the chunks in batches of 100, and store them in ChromaDB with metadata. The Online Pipeline runs on every query: embed the question, run collection.query for the nearest chunks, inject them into a grounded prompt, and generate the answer. Keeping these separate clarifies where cost lives (embedding at ingest, generation at query) and where latency and accuracy trade-offs happen.
How does chunking affect retrieval quality?
Chunking is where most RAG systems succeed or fail. Chunks too small lose context; chunks too large add noise and inflate token costs. The sweet spot is 300–500 tokens per chunk with ~50 characters of overlap so sentences at boundaries are never lost — the window advances by chunk_size minus chunk_overlap each iteration, like overlapping Venn diagrams. As a data scientist, treat chunk size and n_results as hyperparameters: test against real questions, inspect the retrieved chunks and their similarity scores, and calibrate for your document type.
How does semantic search differ from keyword matching under the hood?
Semantic search finds chunks by vector proximity in embedding space, not by exact keyword overlap. That's why a RAG system answers 'how long do I have to return something?' from text that says 'customers can return items within 30 days.' ChromaDB compares the question vector against all stored chunk vectors by coordinate proximity and ranks by similarity. Understanding this lets you reason about why a wrong chunk was retrieved — usually a chunking or embedding-model mismatch, not a search bug.
How do you control hallucinations experimentally?
The single most effective control is the context-only prompt: 'Answer the question using only the context below. If the answer is not in the context, say I do not know.' Run controlled tests — ask questions that are in the document and out of it, and confirm the model cites pages for the former and abstains on the latter. This turns hallucination control from a vague hope into a measurable property you can regression-test as you tune the system.
Next step
Run the dot-product validation script today to build intuition, then implement the Offline and Online pipelines against a single PDF using ChromaDB's PersistentClient — no account needed. Once the fundamentals click, wrap it in FastAPI and Streamlit to see the full stack, then experiment with chunk sizes, n_results, and embedding models to understand each hyperparameter's impact.
// FREQUENTLY ASKED QUESTIONS
What's the fastest way to build intuition for how RAG works?
Write the embedding validation script first: embed a few sentences with text-embedding-3-small and compute dot products between them. Seeing similar sentences score high and unrelated ones score near zero makes embedding space concrete. From there, the two-pipeline architecture and semantic search follow naturally because you understand what the coordinates mean.
Should I start with ChromaDB or a production vector database?
Start with ChromaDB. It runs locally via chromadb.PersistentClient() with zero account or setup, so you focus on RAG concepts instead of infrastructure. Once you understand chunking, embedding, and retrieval, swap in Pinecone, Weaviate, or pgvector for production — the retrieval logic stays the same, only the storage backend changes.
How do I treat chunk size and n_results as tunable hyperparameters?
Test against a fixed set of real questions and inspect the retrieved chunks and similarity scores for each setting. Smaller chunks (300 tokens) boost precision on dense text; larger chunks preserve narrative context. Increase n_results only when answers miss information spread across sections, watching token cost. Calibrate empirically rather than guessing.