Production RAG Pipeline Engineering Framework

// When should you use the Production RAG Pipeline Engineering Framework?

Use this skill whenever you are building, auditing, or scaling a Retrieval Augmented Generation (RAG) system — especially when retrieval quality is poor, the system breaks at scale, or you need to move from a working prototype to a production deployment.

// What do you need before building a production RAG pipeline?

• Document corpusrequired
  The raw files to be indexed (PDF, TXT, HTML, DOCX, CSV, etc.) and their approximate size and content type
• Use case descriptionrequired
  What the RAG system must answer, who the users are, and whether accuracy or speed is the higher priority
• Embedding model choicerequired
  Which embedding model will be used for indexing (e.g., text-embedding-3-small, text-embedding-3-large, BGE-small). Must be identified upfront and kept consistent.
• LLM choicerequired
  The chat model used for generation (e.g., GPT-4o, Claude Sonnet). Separate from the embedding model.
• Vector store choice
  Which vector database will be used (e.g., Chroma, Pinecone, FAISS)
• Quality vs. speed constraint
  Whether the system must prioritise retrieval accuracy (use semantic chunking) or processing throughput (use recursive chunking)

// What core principles govern production RAG engineering?

// How do you build a production RAG pipeline step by step?

1. 1

   Load documents using the appropriate Document Loader

   Select loader based on content type: PyPDFLoader for simple PDFs (fast, basic metadata), PyMuPDFLoader for high-volume PDFs (fastest, best metadata), UnstructuredPDFLoader for complex layouts with tables (slower, richest metadata), TextLoader for plain text, DirectoryLoader for bulk folder ingestion (use glob pattern to filter file types), WebBaseLoader for URLs. Output is a list of LangChain Document objects with page_content and metadata fields (source, page number, author, etc.).

2. 2

   Choose and apply the correct Chunking Strategy

   Apply the Chunking Decision Framework: (1) Is this a quick prototype or simple structured doc? → Use Recursive Chunking. (2) Is quality critical (legal, technical manuals, knowledge bases)? → Use Semantic Chunking. (3) Is it code? → Use the Code Splitter. (4) Is it Markdown? → Use the MD Splitter with header-based chunk size. NEVER use Fixed Size Chunking in production — it destroys meaning by splitting at arbitrary character counts regardless of sentence or concept boundaries. For Recursive Chunking, set chunk_size between 500–1,000 tokens and add overlap (typically 50–200 tokens) to preserve context at boundaries.

3. 3

   Understand the four Chunking Variables and tune them

   Variable 1 — Chunk Size: too small = fragment loss and retrieval noise; too large = diluted meaning and wasted token budget. Sweet spot: 200–1,000 tokens. Variable 2 — Overlap: preserves context at chunk boundaries; no overlap = context loss between chunks. Variable 3 — Split Boundaries: Fixed (random cut, avoid in production) → Recursive (paragraph/sentence/clause hierarchy, reliable default) → Semantic (meaning boundaries, premium quality). Variable 4 — Content Type: code must keep functions together; legal docs need semantic chunking; markdown splits on headers.

4. 4

   Select and lock the Embedding Model before indexing

   Choose based on the quality/speed/cost trade-off. Dimensions indicate semantic capacity: text-embedding-3-small = 1,536 dims (good balance); text-embedding-3-large = 3,072 dims (highest quality, more storage); Gemini embeddings = 768 dims (free, smaller); BGE-small = 384 dims (fastest, lowest semantic nuance). Sweet spot for production: 768–1,536 dimensions. Lock this model before indexing and never change it without re-indexing your entire corpus.

5. 5

   Generate embeddings and populate the Vector Store (Indexing Phase)

   This is a one-time process per corpus update. Each chunk is passed through the embedding model to produce a vector. Vectors, original text chunks, and metadata are stored together in the vector store (e.g., Chroma, Pinecone). Use get_or_create_collection and upsert (not add) so re-runs do not duplicate documents. At the end of indexing, the vector store contains thousands of vectors each representing a chunk of your documents.

6. 6

   Build the RAG Chain with RunnablePassthrough and a grounding prompt

   Construct the chain as: {context: retriever | format_docs, question: RunnablePassthrough()} | prompt_template | llm | StrOutputParser(). The context input comes from the retriever (relevant document chunks). The question uses RunnablePassthrough so the original query reaches the LLM unchanged. The prompt must explicitly state: 'Answer based only on the following context. If the context doesn't contain the answer, say I don't know.' Pass both context and question into the prompt template.

7. 7

   Configure the Retriever with similarity search and metadata filtering

   Instantiate the retriever from the vector store with search_type='similarity' and set k (number of documents to retrieve, typically 2–5). Use metadata filtering to narrow retrieval to relevant subsets (e.g., filter by topic, source, date). Understand score semantics: distance scores (closer to 0 = more relevant) vs. similarity scores (closer to 1 = more relevant). Convert if needed: similarity = 1 / (1 + distance).

8. 8

   Attach source tags to every retrieved chunk

   Use format_docs_with_source to add source tags to each chunk before it enters the prompt. Output format should include the source document name/path alongside the page content. This allows users to verify answers, enables citation, and builds system trust. Sources are essential for production — not optional.

9. 9

   Test retrieval in isolation before evaluating generation

   Run queries against the retriever alone and inspect the raw documents returned. Ask: Are these chunks actually relevant to the query? Do they contain complete information or are they fragments? If retrieval is broken, fix chunking and/or the embedding model before touching the LLM prompt. 90% of RAG failures are retrieval failures.

10. 10

    Identify and remediate the Five Main Failure Modes

    The five failure modes that cause 90% of production RAG failures are: (1) Bad chunking → shattered meaning, incomplete embeddings; (2) Wrong or mismatched embedding models between indexing and querying; (3) Missing or weak retrieval prompt grounding → hallucination; (4) No metadata filtering → irrelevant document retrieval; (5) No source attribution → unverifiable answers. Address each systematically before declaring the system production-ready.

11. 11

    Evaluate advanced architectures for cutting-edge requirements

    Once the baseline RAG is production-stable, consider: Late Chunking (embed full document first, then pool chunk embeddings — 10–12% accuracy improvement, requires models like Jina Embeddings v2); Semantic Chunking (split at meaning boundaries using embedding similarity drops); Agentic RAG (retrieval driven by agent reasoning loops); Graph RAG (knowledge graph-augmented retrieval); Contextual Retrieval; Multimodal RAG (images, tables, mixed media). The verdict: semantic chunking is the practical best for now; late chunking is the future.

// What do real RAG debugging and scaling scenarios look like?

// What are the most common RAG pipeline mistakes to avoid?

• Using Fixed Size Chunking in production — it cuts at arbitrary character counts, destroys meaning, and causes retrieval failure before the LLM is even involved. Never use this in production. Ever.
• Using different embedding models at indexing time vs. query time — vectors become incomparable and similarity search silently fails.
• Blaming the LLM for bad answers without first testing retrieval in isolation — 90% of RAG failures are retrieval failures, not generation failures.
• Skipping the grounding prompt pattern — without explicitly instructing the LLM to answer only from context and say 'I don't know' when unsure, hallucination is guaranteed.
• Omitting source attribution — without sources users cannot verify answers, trust degrades, and the system is not production-ready.
• Chunking code by character count — splitting a for-loop across two chunks makes both chunks semantically useless. Use the Code Splitter and keep functions together.
• Treating chunk size as a fixed universal constant — the right chunk size depends on content type, retrieval use case, and token budget. Technical and legal content often needs semantic chunking with auto-determined sizes.
• Using the add method instead of upsert when populating the vector store — repeated runs will duplicate all documents, corrupting retrieval results.
• Ignoring overlap between chunks — without overlap, context at chunk boundaries is permanently lost and adjacent concepts become disconnected.
• Mixing up embedding models and chat models — embedding models output vectors (not text) and are a completely different model type from chat models like GPT-4o or Claude, even from the same provider.

// What key RAG terms should you know?

RAG (Retrieval Augmented Generation)

An architecture that grounds LLM responses in actual retrieved documents, reducing hallucination by augmenting the model's knowledge with relevant external chunks at query time.

Chunking

The process of splitting documents into smaller pieces before embedding. The single biggest lever over RAG quality — described by the creator as architecture, not pre-processing.

Fixed Size Chunking

Splitting every n characters or tokens regardless of sentence or meaning boundaries. Destroys semantic meaning, causes fragment loss, and should never be used in production.

Recursive Chunking

The reliable default chunking strategy used by LangChain. Uses a split hierarchy — paragraphs → sentences → clauses → words → characters — always preferring natural boundaries over arbitrary cuts.

Semantic Chunking

The premium chunking strategy that splits at meaning boundaries by embedding each sentence and splitting where adjacent embedding similarity drops. Best for legal, technical, and high-value RAG where accuracy > speed.

Late Chunking

An advanced approach that embeds the full document first, then pools embeddings for individual chunks — preserving cross-chunk context. Yields ~10–12% accuracy improvement. Requires models like Jina Embeddings v2. Called 'the future of chunking' by the creator.

Embedding Model

A model that takes text in and outputs a vector (list of numbers representing semantic meaning). Completely different from a chat model. Must be the same model at both indexing and querying phases.

Dimensions

The size of the vector output by an embedding model (e.g., 1,536 for text-embedding-3-small). More dimensions = more semantic nuance captured but more storage and slower search. Sweet spot: 768–1,536.

Indexing Phase

The one-time process of loading documents, chunking them, embedding each chunk, and storing vectors in the vector store. The foundation of any RAG system.

Querying Phase

The per-question process of embedding the user query using the same embedding model, searching the vector store for similar vectors, retrieving original text chunks, and passing them with the query to the LLM.

RunnablePassthrough

A LangChain construct that passes the user's question through the RAG chain unchanged — ensuring the original query reaches the LLM without modification.

format_docs_with_source

A function that adds source tags to each retrieved chunk so the final answer includes citation metadata (document name, page, etc.), enabling user verification.

Overlap

A configurable number of shared tokens between adjacent chunks that preserves context at chunk boundaries and prevents meaning loss at split points.

Grounding Prompt Pattern

The prompt engineering pattern: 'Answer based only on the following context. If the context doesn't contain the answer, say I don't know.' The low-hanging fruit for preventing hallucination that most people ignore.

Five Main Failure Modes

The five reasons 90% of RAG systems fail in production: bad chunking, mismatched embedding models, missing grounding prompt, no metadata filtering, and no source attribution.

Distance Score vs. Similarity Score

Distance scores: closer to 0 = most relevant (used by Chroma by default). Similarity scores: closer to 1 = most relevant. Convert via: similarity = 1 / (1 + distance). Know which your vector store returns.

Agentic RAG

An advanced RAG architecture where an agent reasoning loop drives retrieval decisions dynamically, rather than a fixed retrieval step.

Graph RAG

A RAG architecture that augments retrieval with a knowledge graph to capture entity relationships and multi-hop reasoning across documents.

Multimodal RAG

A RAG architecture that handles mixed content types including images, tables, and text in the same retrieval pipeline.

// FREQUENTLY ASKED QUESTIONS