KodeKloud Complete RAG System Design Skill

// When should you use RAG system design?

Use this skill whenever a user needs to extend an LLM's implicit knowledge with external, proprietary, or domain-specific data that was not part of the model's pre-training. Trigger this skill when the question involves building a document search system, internal chatbot, knowledge-base Q&A, or any architecture where the LLM must retrieve before it generates.

// What do you need before designing a RAG pipeline?

• Use-case descriptionrequired
  What problem is the RAG system solving? (e.g., law firm document search, internal HR chatbot, customer-facing FAQ)
• Data source characteristicsrequired
  Volume, format, and update frequency of the external documents (e.g., 10 GB of PDFs updated monthly, structured HR policies, image-heavy reports)
• Query typerequired
  How users will phrase queries — keyword-style, natural-language questions, multi-part questions, or goal-oriented agent requests
• Latency and cost tolerancerequired
  Whether slower, higher-quality retrieval is acceptable or fast single-shot responses are required
• Modality requirements
  Whether data contains images, charts, diagrams, or tables that must also be searchable
• Hierarchical structure
  Whether documents are organised hierarchically (e.g., company > division > department) and whether that structure must be preserved

// What core principles govern how RAG systems work?

// How do you build a RAG system step by step?

1. 1

   Assess whether RAG is the right tool

   RAG is appropriate when: (a) search must be by semantic meaning rather than exact text; (b) you have large, disparate document sets that need unified search. RAG is NOT appropriate when: (a) you need to search by document format, page position, or visual layout — use a vision model instead; (b) data is primarily images, charts, or graphs without text — vanilla vector embeddings cannot represent non-text modalities unless Multimodal RAG is explicitly added.

2. 2

   Select and configure an embedding model

   Choose an embedding model to convert documents and queries into semantic vectors. Common options: OpenAI text-embedding-3-large (cloud, high quality), Cohere embedding models (cloud), all-MiniLM-L6-v2 (local, lightweight). The same embedding model MUST be used for both document ingestion and query-time retrieval — mismatching models breaks semantic alignment.

3. 3

   Select a vector database

   Choose a vector database to store both the embedding vectors and the original chunk text. Popular options: Chroma (local/dev), Pinecone (managed/production). For access control (e.g., law firm per-case privacy), add metadata filters at storage time so retrieval can be scoped to a specific subset of documents.

4. 4

   Choose and apply a chunking strategy

   Select chunking strategy based on document type and quality requirements: - Fixed-size chunking: Split by character count, word count, or token count. Simplest; use as a baseline. Set chunk_size and chunk_overlap (e.g., 200 chars, 50 char overlap). Use LangChain RecursiveCharacterTextSplitter. - Sentence-aware chunking: Use spaCy to respect sentence boundaries. Better for prose documents. - Overlapping / Sliding Window chunking: Add deliberate overlap between chunks (e.g., 50-char lip) so adjacent chunks share context. 'More like art than science' — tune the overlap empirically. - Semantic chunking: Break text where meaning shifts by measuring sentence-to-sentence similarity. Preserves natural topic boundaries. Higher engineering overhead. - Agentic chunking: Pass documents to an LLM and let it decide optimal split points based on semantic topic shifts. Highest quality; highest cost and latency. Use affordable LLMs to reduce cost. Requires rerunning agents if documents change. For most applications, sentence-aware chunking with overlap is a good balance. For high-value documents where quality matters most, agentic chunking provides the best results.

5. 5

   Ingest documents into the vector database

   For each chunk: (1) run it through the embedding model to produce a vector; (2) store both the vector and the original chunk text in the vector database, along with any metadata (document ID, section, date, access scope). Verify the collection count after ingestion. If documents are hierarchically organised (company > division > department), consider Hierarchical RAG — store summary embeddings at each level so coarse-level retrieval happens before fine-grained retrieval.

6. 6

   Build the retrieval pipeline

   At query time: (1) embed the user's query using the same embedding model; (2) run a vector similarity search (cosine similarity is the standard approach) against the vector database; (3) retrieve the top-K most similar chunks; (4) pass the retrieved chunks as context into the LLM prompt; (5) the LLM generates its answer from the retrieved context. The retrieved data is ephemeral — it exists only for this turn.

7. 7

   Select the appropriate RAG variant for the use case

   Choose based on latency tolerance and use-case complexity: - Standard RAG (single-shot): One query → retrieve → generate. Use for most applications. - Cache Augmented Generation (CAG): Add a cache layer checked before the vector database. Best for content that does not change often. Invalidate cache when underlying data changes. - Multi-Query RAG: Use an LLM to generate multiple rephrased variants of the original query, run each through RAG, then merge and deduplicate results. Casts a wider net. Slower and potentially noisier; best for generic, exploratory queries. - Agentic RAG: The agent formulates its own retrieval strategy as a goal-based system rather than a task-based single shot. Slower than standard RAG; use only when slower responses are tolerated in exchange for potentially higher quality data. - Hierarchical RAG: Preserves the hierarchical structure of corporate documents. Retrieval checks coarse levels before fine levels. Adds engineering overhead in setup and maintenance. - Multimodal RAG: Extends beyond text to images, charts, diagrams, and screenshots by converting all modalities into a shared embedding space. Required when documents contain non-text content that must be retrievable.

8. 8

   Evaluate the RAG system with retrieval metrics

   Define ground truth data: a set of test queries each mapped to the document IDs that should be retrieved. Then measure: - Precision at K: (number of relevant docs in top K) / K. Use when you want to minimise noise in results. - Recall at K: (relevant docs found in top K) / (total relevant docs that exist). Use when missing a document is unacceptable (e.g., legal evidence). - Mean Reciprocal Rank (MRR): 1 / (position of the first relevant document). Use for Q&A systems where only the top result matters. Position 1 = MRR 1.0; Position 2 = 0.5; Position 3 = 0.33. - Normalized Discounted Cumulative Gain (NDCG): Rewards placing relevant documents at higher positions; penalises burying them lower. NDCG = 1.0 means perfect ranking. Use for search engines where the order of all results matters. Your specific setup may require measuring different segments — choose the metric that matches the cost of failure for your use case.

9. 9

   Iterate and harden

   Use metric results to diagnose failure modes: low Precision at K → chunks are too broad or retrieval top-K is too large; low Recall at K → chunking is losing context at boundaries, try overlap or semantic chunking; low MRR → the most relevant chunk is being out-ranked, review embedding model quality or chunk granularity; low NDCG → ranking order is wrong, review similarity scoring. Re-chunk, re-embed, and re-evaluate until metrics meet the use-case threshold.

// What do real-world RAG implementations look like?

// What mistakes should you avoid when building RAG systems?

• Believing RAG gives LLMs long-term memory — retrieved data is ephemeral (persists only during the turn); the persistence illusion comes from the database being available, not from the model retaining anything.
• Storing entire documents as a single chunk — this returns the entire document on any query match, floods the context window with irrelevant content, and defeats the purpose of semantic search.
• Using different embedding models for ingestion and query time — semantic alignment breaks entirely if the models differ.
• Using RAG when the search requirement is by document format, page location, or visual layout — RAG retrieves by semantic meaning only, not by position or physical structure; use a vision model for layout-based retrieval.
• Applying RAG to image, chart, or graph search without Multimodal RAG — vanilla vector embeddings only represent text; non-text modalities require a shared embedding space.
• Choosing fixed-size chunking without considering semantic boundaries — abruptly splitting a document by character count can cut across a coherent idea, degrading retrieval quality.
• Setting chunk overlap arbitrarily — overlap amount is 'more like art than science'; too little loses boundary context, too much creates excessive redundancy and storage cost.
• Deploying Agentic Chunking without accounting for cost and reprocessing overhead — any change to documents requires rerunning the LLM-based chunking agent across the entire affected corpus.
• Deploying Agentic RAG or Multi-Query RAG when latency is critical — both variants are slower than single-shot RAG by design and should only be used when slower responses are tolerated.
• Using CAG (Cache Augmented Generation) on frequently changing data — if the underlying dataset changes faster than the cache is invalidated, the model answers from stale cached results.
• Evaluating a RAG system with only one metric — Precision, Recall, MRR, and NDCG each measure a different failure mode; relying on a single metric will miss systemic problems.

// What are the key RAG terms you need to know?

Retrieval Augmented Generation (RAG)

A method that allows an LLM to augment its implicit knowledge by retrieving relevant chunks from an external data source and generating its answer from those retrieved chunks, rather than relying solely on what was learned during pre-training.

Implicit knowledge

The knowledge an LLM absorbed during pre-training from trillions of tokens. It cannot be directly inspected or updated without retraining.

Vector database

A specialised database (e.g., Chroma, Pinecone) that stores document chunks alongside their vector embeddings, enabling similarity-based semantic search rather than exact keyword matching.

Embedding model

A model (e.g., OpenAI text-embedding-3-large, Cohere, all-MiniLM-L6-v2) that converts raw text into a high-dimensional vector — a semantic representation — so that meaning rather than literal text can be searched.

Context window

The hard limit on how many tokens an LLM can process in a single turn. All retrieved chunks must fit within this window; exceeding it causes the model to refuse or truncate the input.

Chunking

The process of splitting a document into smaller, semantically coherent pieces before embedding and storing them in a vector database. The most impactful engineering decision in a RAG pipeline.

Fixed-size chunking

Splitting a document by a predefined number of characters, words, tokens, or sentences. Simplest strategy but ignores semantic grouping and can abruptly cut through coherent ideas.

Overlapping chunking (Sliding Window)

A chunking method that intentionally adds a 'lip' of duplicated text between adjacent chunks so each chunk retains some context from the previous and next chunk. Described as 'more like art than science' in tuning the overlap amount.

Semantic chunking

Splitting a document at points where the meaning shifts, measured by dropping sentence-to-sentence similarity. Preserves natural topic boundaries at the cost of higher engineering overhead.

Agentic chunking

Delegating the chunking decision to an LLM, which analyses the document and decides optimal split points based on semantic topic shifts. Highest quality; highest cost and latency; requires rerunning agents when documents change.

Ephemeral retrieval

The property that data retrieved by RAG exists only during the current turn and is not persisted by the LLM itself. This is why RAG does not provide true long-term memory.

Precision at K

A retrieval metric: (number of relevant documents in the top K results) / K. Measures the quality — i.e., how much noise — is in the retrieved set.

Recall at K

A retrieval metric: (relevant documents found in top K) / (total relevant documents that exist). Measures coverage — how many of the documents that should have been found actually were.

Mean Reciprocal Rank (MRR)

A retrieval metric: 1 / (position of the first relevant document in the result list). Measures how quickly a user encounters the first useful result. Best suited for Q&A systems.

Normalized Discounted Cumulative Gain (NDCG)

A retrieval metric that evaluates the full ranking by rewarding relevant documents placed at higher positions and penalising them when buried lower. NDCG = 1.0 means perfect ranking. Best suited for search engines.

Cache Augmented Generation (CAG)

A RAG variant that adds a cache layer checked before the vector database. If the cache contains a sufficiently relevant prior answer, it is returned without hitting the RAG pipeline. Best for content that changes infrequently.

Agentic RAG

A RAG variant where an AI agent formulates its own retrieval strategy as a goal-based system rather than executing a single-shot retrieval. Slower than standard RAG; used when slower responses are tolerated in exchange for potentially higher quality data.

Multi-Query RAG

A RAG variant where an LLM generates multiple rephrased variants of the original user query, each is run through the RAG pipeline independently, and results are merged and deduplicated before generation. Casts a wider net; slower and potentially noisier.

Hierarchical RAG

A RAG variant that preserves the natural hierarchical structure of documents (e.g., company > division > department) by storing summary embeddings at each level and retrieving coarse-to-fine rather than flattening all content into a single embedding space.

Multimodal RAG

A RAG variant that extends retrieval beyond text to include images, charts, diagrams, and screenshots by converting all modalities into a shared embedding space so non-text content can be semantically searched and retrieved.

Cosine similarity

The standard mathematical approach for comparing two vectors in a vector database to identify how semantically similar they are. The closer the cosine similarity score to 1, the more semantically related the two pieces of text.

Ground truth data

A set of test queries each mapped to the document IDs that should ideally be retrieved. Ground truth is essential for running Precision at K, Recall at K, MRR, and NDCG evaluations.

// FREQUENTLY ASKED QUESTIONS