Frequently Asked Questions About KodeKloud Complete RAG System Design Skill
22 answers covering everything from basics to advanced usage.
// Basics
What is the difference between semantic search and keyword search in RAG?
Keyword search (TF-IDF, BM25) matches exact terms and fails when queries use synonyms or rephrasing. Semantic search uses embedding models to retrieve by meaning, so a query for 'distributed workforce policies' can surface documents about 'remote work policy' with zero word overlap. RAG relies on semantic search, which is why documents are stored as embedding vectors representing meaning rather than as raw searchable text.
What is the context window and how does it constrain my RAG design?
The context window is the hard limit on how many tokens an LLM can process in a single turn. Every chunking and retrieval decision must respect it: your retrieved top-K chunks combined must fit inside the target LLM's context window. Exceeding it causes truncation or refusal. This constraint drives chunk size, top-K choice, and why storing whole documents as single chunks fails.
What inputs do I need to gather before starting a RAG project?
You need a use-case description (the problem the system solves), data source characteristics (volume, format, update frequency), query type (keyword, natural-language, multi-part, or agent goals), and latency and cost tolerance. Optionally, gather modality requirements (images, charts, tables) and hierarchical structure (company > division > department). These inputs determine chunking strategy, RAG variant, embedding model, and which retrieval metric matters most.
// How To
How do I set chunk size and overlap correctly?
Start with a baseline like chunk_size 200 characters and chunk_overlap 50 using LangChain's RecursiveCharacterTextSplitter, then tune empirically. Overlap is 'more like art than science': too little loses boundary context, causing low Recall; too much creates redundancy and storage cost. Adjust based on metric feedback — if Recall at K is low, increase overlap or switch to semantic chunking to stop losing context at boundaries.
How do I add access control so users only see documents they're allowed to?
Add metadata filters at storage time. When ingesting each chunk, attach fields like matter-ID, access scope, or department, then apply a filter at retrieval time so vector similarity search is scoped to only permitted documents. For a law firm, tag each chunk with a matter-ID and filter by it at query time to prevent cross-matter data leakage.
How do I ingest documents into a vector database?
For each chunk: run it through the embedding model to produce a vector, then store both the vector and the original chunk text in the vector database along with metadata (document ID, section, date, access scope). Verify the collection count after ingestion to confirm all chunks landed. If documents are hierarchical, also store summary embeddings at each level for coarse-to-fine retrieval.
How do I choose which retrieval metric to optimize for?
Match the metric to the cost of failure. Use Precision at K to minimize noise; Recall at K when missing a document is unacceptable, like legal evidence; MRR for Q&A systems where only the top result matters; and NDCG for search engines where the order of all results matters. No single metric captures every failure mode, so measure a suite and diagnose accordingly.
What embedding models should I consider and how do I choose?
Common options are OpenAI text-embedding-3-large (cloud, high quality), Cohere embedding models (cloud), and all-MiniLM-L6-v2 (local, lightweight). Choose based on quality needs, cost, latency, and whether you can run local for privacy. Whatever you pick, use the exact same model for both ingestion and query-time retrieval, since mismatching models breaks semantic alignment. If MRR is low, upgrading the embedding model often improves ranking quality.
// Troubleshooting
My RAG system has low Recall at K — how do I fix it?
Low Recall means chunking is losing context at boundaries, so relevant documents aren't surfacing. Add overlap between chunks or switch to semantic chunking to preserve topic boundaries. You can also increase top-K, but that risks lowering Precision. Re-chunk, re-embed, and re-evaluate against your ground truth until Recall meets your threshold, especially critical when missing a document is unacceptable.
My RAG returns lots of irrelevant results — what's wrong?
That's low Precision at K, meaning chunks are too broad or your retrieval top-K is too large. Narrow chunk size, reduce top-K, or use more precise chunking that avoids merging unrelated topics. Also verify you're using the same embedding model for ingestion and querying — a mismatch produces near-random results. Re-evaluate Precision after each change.
Why is my RAG answering with outdated information?
If you're using Cache Augmented Generation (CAG) on frequently changing data, the model may be answering from stale cached results. CAG is only safe for content that changes infrequently, like a quarterly HR handbook, and you must invalidate the cache whenever underlying data changes. If data changes faster than cache invalidation, remove the cache layer and query the vector database directly.
The most relevant chunk keeps getting out-ranked — how do I fix ranking?
That's a low MRR or low NDCG problem. Low MRR means your best chunk is being out-ranked, so review embedding model quality or chunk granularity — a higher-quality embedding model often lifts ranking. Low NDCG means overall ranking order is wrong, so review your similarity scoring. Confirm cosine similarity is applied correctly, then re-embed and re-evaluate against ground truth.
// Comparisons
How does Standard RAG compare to Agentic RAG?
Standard RAG is single-shot: one query, retrieve, generate — fast and sufficient for most applications. Agentic RAG lets an AI agent formulate its own retrieval strategy as a goal-based system rather than a single task, potentially producing higher-quality data. The trade-off is speed: Agentic RAG is slower by design. Use it only when slower responses are acceptable in exchange for retrieval quality.
When should I use Multi-Query RAG instead of Standard RAG?
Use Multi-Query RAG for generic, exploratory queries where coverage matters more than speed. An LLM generates multiple rephrased variants of the original query, each runs through RAG independently, and results are merged and deduplicated to cast a wider net. It's slower and potentially noisier than Standard RAG, so optimize for Recall at K. For precise, latency-sensitive queries, stick with Standard RAG.
What is the difference between CAG and a traditional RAG cache?
Cache Augmented Generation (CAG) adds a cache layer checked before the vector database: if a sufficiently relevant prior answer exists, it's returned without hitting the RAG pipeline, cutting latency and cost. It's best for content that changes infrequently. The critical rule is cache invalidation — you must clear the cache whenever underlying data changes, or the system will serve stale answers.
How does fixed-size chunking compare to semantic and agentic chunking?
Fixed-size chunking splits by character, word, or token count — simplest but can cut across coherent ideas, degrading retrieval. Semantic chunking splits where meaning shifts by measuring sentence similarity, preserving topic boundaries with more engineering overhead. Agentic chunking delegates split decisions to an LLM for the highest quality, at the highest cost and latency, and requires rerunning agents when documents change. Sentence-aware with overlap is a solid default.
// Advanced
When is RAG the wrong tool for the job?
RAG is wrong when search must be by document format, page position, or visual layout — use a vision model instead. It's also wrong when data is primarily images, charts, or graphs without text, since vanilla embeddings only represent text unless you explicitly add Multimodal RAG. And it's wrong if you actually need conversational long-term memory, since RAG retrieval is ephemeral, not persistent.
How does Hierarchical RAG work and when should I use it?
Hierarchical RAG preserves a document's natural structure (company > division > department) by storing summary embeddings at each level and retrieving coarse-to-fine rather than flattening everything into one embedding space. Retrieval checks coarse levels before fine levels. Use it when structure carries meaning and must be preserved, such as legal documents organized by matter > filing date > document type. It adds setup and maintenance overhead.
How do I make images and charts searchable in a RAG system?
Use Multimodal RAG, which extends retrieval beyond text to images, charts, diagrams, and screenshots by converting all modalities into a shared embedding space. This lets non-text content be semantically searched alongside text. Vanilla vector embeddings can't represent non-text modalities, so applying standard RAG to image-heavy data without Multimodal RAG is a common pitfall that yields no useful retrieval for those assets.
What are the hidden costs of agentic chunking at scale?
Agentic chunking passes documents to an LLM to decide optimal split points, so cost scales with corpus size and LLM pricing. The bigger hidden cost is reprocessing: any change to documents requires rerunning the chunking agent across the entire affected corpus, since split decisions depend on full-document analysis. Use affordable LLMs to reduce cost, and reserve agentic chunking for high-value documents where quality justifies the overhead.
How do I build reliable ground truth data for evaluation?
Ground truth is a set of test queries each mapped to the document IDs that should ideally be retrieved. Build it by sampling representative real queries and having domain experts label which documents are genuinely relevant to each. This dataset is essential for running Precision, Recall, MRR, and NDCG. Keep it current as documents change, and segment it by query type if different segments have different failure costs.
Can I combine multiple RAG variants in one system?
Yes, and it's common. For example, a law firm system can combine Standard RAG with per-matter metadata filters and add Hierarchical RAG when documents are organized by matter > filing date > document type. An HR chatbot can combine Standard RAG with a CAG layer for infrequently changing policies. Choose variants by matching each to a specific constraint — latency, structure, modality, or coverage — rather than defaulting to complexity.