How to Design a Production RAG Pipeline
For ML engineers building RAG systems · Based on Tejas AI Agentic AI Builder Framework
// TL;DR
ML engineers use the Tejas AI Agentic AI Builder Framework's RAG methodology to build retrieval systems that ground answers in private or recent knowledge without hallucination. The framework specifies three phases — Index, Retrieve, Generate — plus the non-negotiable rule to use the same embedding model for both indexing and querying. It goes further with hierarchical chunking for the precision-context balance, cosine similarity and HNSW for fast search, and Agentic RAG so the agent can iterate retrieval when the first pass is insufficient. Use it when your agent needs access to documents outside its training data.
What are the three phases of a RAG pipeline?
RAG works in three phases. Indexing: chunk your documents, embed each chunk with a chosen embedding model, and store the vectors in a vector database. Retrieval: embed the incoming user query, run a similarity search (cosine similarity for text), and pull the top-k most relevant chunks. Generation: inject those retrieved chunks into the LLM prompt alongside the question to produce a grounded answer.
This structure eliminates hallucination about unknown or outdated information because the model reasons over retrieved facts instead of guessing from parametric memory. It's how a legal firm answers questions across hundreds of internal PDFs no LLM was trained on.
What's the one RAG rule I can't break?
Always use the same embedding model for indexing and querying. If you index with one model and query with another, the vectors live in different mathematical spaces — they become completely incompatible and your retrieval results will be garbage. This is the single most common cause of a RAG system that 'works in the notebook but returns nonsense in production.' Pin the embedding model version and treat it as part of your data contract.
How do I get both precision and context in retrieval?
Use hierarchical chunking. Store small precise chunks for accurate retrieval and larger parent chunks for full context. When a small chunk matches the query, you retrieve its parent to give the model complete surrounding context. This combines the precision of small chunks — which match specific queries tightly — with the contextual completeness of large chunks that the model needs to reason well.
Under the hood, most vector databases use HNSW (Hierarchical Navigable Small World) indexing to find approximate nearest neighbors fast, navigating a smart graph instead of comparing every stored vector. That trades negligible accuracy for massive speed, which is what makes production-scale retrieval feasible.
When should I upgrade to Agentic RAG?
Upgrade to Agentic RAG when a single fixed retrieval pass isn't enough for complex, multi-part questions. In Agentic RAG, the agent treats retrieval as a tool it can call multiple times — if the first results are insufficient, it refines the query and retrieves again, deciding when and what to retrieve rather than following a rigid pipeline. This is ideal for legal, research, or technical questions where the first hit rarely covers everything.
For the vector store itself, prototype in Chroma (5-minute local setup) and move to Pinecone or Qdrant for production-grade filtering and scale.
How do I keep RAG cost and latency under control?
Don't load your entire knowledge base into context — apply the Iceberg Technique. Keep only core rules in the context window and let the agent surgically retrieve additional content with read/grep-style tools. Remember that a larger context window isn't always better: more tokens mean higher cost, slower responses, and potential loss of focus. Tune top-k carefully, and route retrieval summarization to mid-tier models under the 60-30-10 rule, reserving top-tier models for final synthesis.
Next step: Audit your current pipeline for the embedding-model consistency rule, add hierarchical chunking, and prototype an Agentic RAG loop on your hardest multi-part query. Measure retrieval relevance before and after.
// FREQUENTLY ASKED QUESTIONS
Why does my RAG system return irrelevant results?
The most common cause is using different embedding models for indexing and querying, which puts vectors in incompatible spaces. Fix that first. If models match, check your chunking strategy (add hierarchical chunking), confirm you're using cosine similarity for text, tune top-k, and consider Agentic RAG so the agent can retry retrieval with a refined query when results are weak.
What chunking strategy gives the best retrieval accuracy?
Hierarchical chunking gives the best precision-context balance. Store small precise chunks so queries match tightly, and larger parent chunks so the model gets full surrounding context when a small chunk hits. This avoids the trade-off between tiny chunks that lack context and huge chunks that dilute relevance — you get both accuracy and completeness.
When is Agentic RAG worth the extra complexity?
Agentic RAG is worth it for complex, multi-part questions where a single fixed retrieval pass misses key context. Because the agent treats retrieval as a tool and can iterate with refined queries, it handles legal, research, and technical questions far better. For simple single-fact lookups, standard RAG is cheaper and sufficient.