Grigorev RAG Application Build Framework

Last updated: 17 July 2026

Build a working Retrieval Augmented Generation (RAG) application from scratch that answers domain-specific questions using a private knowledge base — without retraining any LLM.

// TL;DR

The Grigorev RAG Application Build Framework is a step-by-step method for building a Retrieval Augmented Generation (RAG) app that answers questions from your private data — without retraining any LLM. It reduces every RAG system to three sequential steps: Search (retrieve relevant documents), Build Prompt (assemble context plus the question), and LLM (generate the answer). Use it whenever you need an LLM to answer from internal, private, or domain-specific data it was never trained on — company FAQs, documentation, course content, or knowledge bases. It also covers modular code architecture, persistent storage, and separating ingestion from the assistant for production.

Framework

// When should you use the Grigorev RAG Application Build Framework?

Use this skill whenever a user needs to make an LLM answer questions from internal, private, or domain-specific data that the LLM was never trained on — such as company FAQs, documentation, course content, or knowledge bases.

// What do you need before building a RAG application?

Knowledge Base / Data Sourcerequired
The collection of documents, FAQs, or records the system should search. Can be a JSON endpoint, database, or file. Must be machine-readable.
User Queryrequired
The question a user wants answered from the knowledge base.
LLM Provider + API Keyrequired
Access to an LLM (e.g. OpenAI, Anthropic, Gemini, Groq). The framework is provider-agnostic; OpenAI is used as the default example.
Course or Domain Filter (optional)
A keyword field value used to restrict search results to a specific sub-domain (e.g. a specific course name, product line, or department).

// What are the core principles behind building a RAG system?

RAG Flow

Every RAG system has exactly three steps in sequence: Retrieval (search the knowledge base), Build Prompt (assemble context + question into a prompt), and LLM (send the prompt to the model and return the answer). Never conflate or skip steps.

LLM as Black Box

Treat the LLM as a black box. You do not need to understand its internals — you only need to learn how to feed it the right inputs and read its outputs. The skill of RAG engineering is in what you put into the LLM, not how the LLM works.

Retrieval = Search

Retrieval and search are synonyms. The R in RAG is simply: search your knowledge base for documents that are likely to contain the answer, then pass only those documents to the LLM. This is cheaper, faster, and more accurate than sending all your data every time.

Augmented Generation

You augment the LLM's generation capability by injecting retrieved context into the prompt. The LLM figures out what within that context is actually relevant — your job is to ensure the relevant answer is somewhere in what you retrieved.

Instructions vs. User Prompt (Two-Part Prompt)

Every prompt has two parts: Instructions (system prompt — constant, never changes, tells the LLM how to behave) and User Prompt / Template (changes with every request, contains the user's question and the retrieved context). Keep these separate in code.

Text Fields vs. Keyword Fields

In your search index, text fields are fields used for ranked full-text search (question, answer, section). Keyword fields are used for exact-match filtering (e.g. course name). Use keyword fields to restrict the search space before ranking, so results from irrelevant sub-domains are excluded entirely.

Boosting

You can assign importance weights to fields during search. A field with boost=2 is twice as important as a field with boost=1 when ranking results. Use boosting to prioritise question-field matches over answer-field matches, for example.

In-Memory vs. Persistent Storage

In-memory search (e.g. minsearch) loses all indexed data when the process stops — you must re-ingest every time. Persistent storage (e.g. SQLite, Elasticsearch, Postgres) keeps data on disk across sessions. For anything beyond a quick prototype, separate the ingestion process from the RAG assistant process and connect them through a persistent database.

Ingestion Process Separation

In production RAG systems, the ingestion process (fetching, cleaning, and indexing documents into the knowledge base) runs independently from the RAG assistant (which queries the knowledge base and generates answers). They are connected only through the shared persistent database.

Modular Code Architecture

Write each RAG component (search, build_prompt, llm) as a separate, swappable function or class method. This means you can replace minsearch with Elasticsearch, or OpenAI with Anthropic, by changing one function — not rewriting the entire system.

// How do you build a RAG application step by step?

1
Prepare and access your knowledge base data
Get your documents into a machine-readable format (JSON, database, API endpoint). Each document should have fields like: question, answer, section, and a keyword field (e.g. course, product, category). Expect to spend significant time on data cleaning and preparation in real projects — this step is routinely underestimated. LLMs can assist with data cleaning.
2
Choose and configure your search engine (index)
For small/prototype projects: use minsearch (in-memory, no dependencies, drop-in). For persistent or larger projects: use SQLite search, Elasticsearch, or a vector database. Define which fields are text fields (searchable) and which are keyword fields (exact-match filters). Fit/build the index with your documents.
3
Implement the Search function
Write a search(question) function that: (1) sends the user query to the index, (2) applies a keyword filter to restrict results to the relevant sub-domain, (3) applies boost weights to prioritise more important fields (e.g. question field gets boost=2), (4) returns top N results (e.g. top 5). This is the R in RAG.
4
Implement the Build Prompt function
Write a build_prompt(question, search_results) function with two components: (a) Instructions — a constant system prompt telling the LLM its task (e.g. 'Answer questions from course participants based on the provided context. If the answer is not found in the context, respond with I don't know.'). (b) User Prompt Template — a template string that formats the user's question and the retrieved context into a single string. Convert each retrieved document dict into a readable string (build_context), then inject into the template.
5
Implement the LLM function
Write an llm(instructions, user_prompt, model) function that: (1) constructs a message history with role=system/developer for instructions and role=user for the user prompt, (2) calls your LLM provider's API, (3) extracts and returns the text response. Track token usage (input tokens, output tokens) to monitor cost. The responses API is preferred over chat completions for OpenAI.
6
Assemble the RAG flow function
Write a rag(question) function that chains all three steps: results = search(question) → prompt = build_prompt(question, results) → answer = llm(instructions, prompt) → return answer. This is the complete RAG pipeline.
7
Refactor into modular, swappable classes
Extract all logic into separate files: ingest.py (load_data, build_index) and rag_helper.py (RAGBase class). Use a class with __init__ accepting index and llm_client as dependencies — not global variables. This enables subclassing to swap search backends or LLM providers without rewriting the core logic. Example: create LlamaRAG(RAGBase) and override only the llm() method.
8
Separate ingestion from the RAG assistant (for production)
Run two independent processes: (1) Ingestion process — fetches data from source, indexes it into the persistent database. (2) RAG assistant — connects to the same persistent database, performs search and generation on user queries. Connect them only through the shared persistent database (SQLite, Postgres, Elasticsearch, etc.). This means indexing happens once (or on a schedule), not on every application start.

// What are real-world examples of RAG applications?

A SaaS company wants to build an internal chatbot that answers employee HR questions from a policy document library.

Step 1: Export HR policy documents as JSON with fields: question/heading, answer/body, section, department (keyword field). Step 2: Index in SQLite search with text fields = [question, answer, section] and keyword field = department. Step 3: Search filters by department='HR' and boosts question field 2x. Step 4: Build prompt with instructions: 'Answer HR questions based only on the provided policy context. If not found, say I don't know.' Step 5: LLM call returns the answer. Run ingestion separately on a nightly schedule; the chatbot process only queries the persistent database.

An e-commerce platform wants to answer customer product questions from a product FAQ database of 5,000 entries.

Step 1: Load product FAQs via API with fields: question, answer, category (keyword). Step 2: Build a minsearch index for prototyping (swap to Elasticsearch later using modular architecture). Step 3: Filter by category matching the product page the user is on; boost question field. Step 4: Assemble a two-part prompt: instructions (constant) + user prompt template (question + top 5 retrieved FAQs as context). Step 5: LLM generates the answer. Monitor token usage per query to track costs.

// What mistakes should you avoid when building a RAG system?

Sending all documents to the LLM instead of retrieving a small relevant subset — this is expensive, slow, and causes the LLM to produce lower quality answers because it gets confused by irrelevant content.
Underestimating data preparation time — in real projects, getting data into clean, machine-readable format is the hardest and most time-consuming step. The example in this framework uses pre-cleaned data; expect significant effort in production.
Using global variables for index and LLM client instead of encapsulating them as class dependencies — this makes it very hard to swap search backends or LLM providers later.
Using in-memory search (minsearch) in production — all indexed data is lost when the process stops, requiring full re-ingestion on every restart. Use persistent storage for anything beyond a quick prototype.
Mixing ingestion logic and RAG assistant logic in the same process — in production, these should be separate processes connected through a persistent database.
Not separating Instructions (system prompt) from the User Prompt Template — treating the whole prompt as one string makes it harder to maintain, update instructions, and swap templates per use case.
Ignoring token usage tracking — without monitoring input/output tokens, you cannot estimate costs or optimise for efficiency.
Prompt injection risk: users can attempt to override instructions via crafted inputs ('ignore all previous instructions'). Do not expose RAG systems with NDA-protected or sensitive data publicly without guard rails on both input and output.

// What are the key terms in RAG engineering?

RAG (Retrieval Augmented Generation): The most common LLM application pattern in industry. It augments an LLM's generation capability by first retrieving relevant documents from a knowledge base and injecting them into the prompt, allowing the LLM to answer questions about data it was never trained on.
RAG Flow: The three-step pipeline that defines every RAG system: (1) Search/Retrieval, (2) Build Prompt, (3) LLM/Generation.
Retrieval: Synonymous with search. The process of querying a knowledge base with the user's question to fetch the top N documents most likely to contain the answer.
Knowledge Base: The searchable repository of documents (FAQs, policies, records) that the RAG system retrieves from. Must be indexed for efficient search.
Ingestion: The process of fetching data from its source, cleaning it, and indexing it into the knowledge base. Should run as a separate process from the RAG assistant in production.
Instructions: The constant, unchanging part of the prompt (system prompt) that tells the LLM how to behave. Example: 'Answer questions from course participants based on the provided context. If the answer is not found in the context, respond with I don't know.'
User Prompt Template: The variable part of the prompt that changes with every request. It is a template that formats the user's question and the retrieved context documents into a string sent to the LLM.
Text Fields: Fields in the search index used for ranked full-text search (e.g. question, answer, section). Terminology borrowed from Elasticsearch.
Keyword Fields: Fields in the search index used for exact-match filtering to restrict the search space (e.g. course name, department). Equivalent to a SQL WHERE clause. Terminology borrowed from Elasticsearch.
Boosting: Assigning a numerical importance weight to a field during search. A boost of 2 on the question field means a keyword match in the question is twice as valuable as a match in a field with boost=1.
minsearch: A lightweight, in-memory Python search library created by Grigorev for educational use and small projects. Uses Python dictionaries internally. Drop-in for Elasticsearch in small datasets. Not persistent — data is lost when the process stops.
SQLite search: A persistent search library (also by Grigorev) that wraps SQLite's FTS5 full-text search capability. Same interface as minsearch but persists data to disk. Used to demonstrate the separation of ingestion and RAG assistant processes.
LLM as Black Box: The design principle that you do not need to understand how the LLM works internally. You only need to know how to format inputs (prompts) and read outputs. RAG engineering skill lies entirely in prompt and retrieval design.
RAGBase: The suggested base class name for the modular RAG implementation. Contains search(), build_context(), build_prompt(), llm(), and rag() as methods with injectable dependencies (index, llm_client), enabling subclassing to swap components.
Persistent Storage: A database that retains indexed data on disk across process restarts (e.g. SQLite, Postgres, Elasticsearch). Contrasted with in-memory storage like minsearch where data disappears when the process ends.
Guard Rails: Validation checks applied to LLM inputs (to catch prompt injection attacks) or LLM outputs (to verify the response is appropriate before returning it to the user).

// FREQUENTLY ASKED QUESTIONS

What is Retrieval Augmented Generation (RAG)?

RAG is an LLM application pattern that answers questions using your private data by first retrieving relevant documents from a knowledge base, then injecting them into the prompt. This lets the LLM answer questions about data it was never trained on. It's the most common LLM application pattern in industry because it's cheaper and more accurate than sending all your data or retraining a model.

What are the three steps of a RAG system?

Every RAG system has exactly three sequential steps: Retrieval (search your knowledge base for relevant documents), Build Prompt (assemble the retrieved context plus the user's question into a single prompt), and LLM (send the prompt to the model and return the answer). Never conflate or skip these steps — this sequence defines the entire RAG flow.

How do I build a RAG application from scratch?

Prepare your data as machine-readable documents, configure a search index (minsearch for prototypes, Elasticsearch or SQLite for production), then write three functions: search() to retrieve documents, build_prompt() to combine context and question, and llm() to generate the answer. Chain them into a rag() function. Finally, refactor into swappable classes and separate ingestion from the assistant for production.

How do I make an LLM answer questions about my company's internal data?

Use RAG: index your internal documents (FAQs, policies, docs) into a searchable knowledge base, then retrieve the most relevant ones for each user query and inject them into the LLM's prompt. You don't retrain the LLM — you feed it the right context. Add a keyword filter (e.g. department) to restrict search to the relevant sub-domain and instruct the LLM to say 'I don't know' if the answer isn't in the context.

How does RAG compare to fine-tuning an LLM?

RAG injects relevant documents into the prompt at query time without touching the model, while fine-tuning retrains the model on your data. RAG is cheaper, faster to update (just re-index new documents), and easier to audit since answers come from retrieved context. Fine-tuning changes model behavior but requires retraining for every data update. For answering questions from a private knowledge base, RAG is almost always the better first choice.

When should I use RAG instead of just prompting the LLM directly?

Use RAG whenever the answer lives in data the LLM was never trained on — internal FAQs, documentation, course content, or product databases. Direct prompting fails because the LLM doesn't know your private information and will hallucinate. RAG retrieves the actual documents and grounds the answer in them, so the LLM works from facts rather than guesses.

What is the difference between text fields and keyword fields in RAG search?

Text fields (like question, answer, section) are used for ranked full-text search where the engine scores relevance. Keyword fields (like course name or department) are used for exact-match filtering to restrict the search space before ranking — equivalent to a SQL WHERE clause. Use keyword fields to exclude irrelevant sub-domains entirely, then rank remaining results by text-field matches.

What results can I expect from building a RAG application?

You'll get an app that answers domain-specific questions accurately from your private knowledge base, without model retraining. Answers are grounded in retrieved documents, reducing hallucination. It's cheaper and faster than sending all data to the LLM every query. With modular architecture, you can swap search backends or LLM providers by changing one function, and monitor token usage to control costs.

What is minsearch and when should I use it?

minsearch is a lightweight, in-memory Python search library created by Alexey Grigorev for education and small projects. It's a drop-in replacement for Elasticsearch on small datasets with no dependencies. Use it for quick prototypes only — it loses all indexed data when the process stops, requiring full re-ingestion on every restart. For anything beyond a prototype, switch to persistent storage like SQLite, Postgres, or Elasticsearch.

How do I prevent prompt injection attacks in a RAG system?

Add guard rails on both inputs and outputs. Users can craft inputs like 'ignore all previous instructions' to override your system prompt, so validate incoming queries and check outputs before returning them. Never expose a RAG system containing NDA-protected or sensitive data publicly without these safeguards. Keeping instructions separate from the user prompt also makes it easier to detect and contain injection attempts.

Why should I retrieve documents instead of sending all my data to the LLM?

Sending all documents is expensive, slow, and produces lower-quality answers because irrelevant content confuses the LLM. Retrieval sends only the small subset of documents likely to contain the answer, which is cheaper (fewer tokens), faster, and more accurate. Your job is to ensure the relevant answer is somewhere in what you retrieve — the LLM figures out what's actually relevant within that context.

How do I structure a prompt for a RAG application?

Split every prompt into two parts: Instructions (the constant system prompt telling the LLM how to behave, e.g. 'Answer based only on the provided context; if not found, say I don't know') and a User Prompt Template (the variable part containing the user's question and retrieved context). Keep these separate in code so you can update instructions or swap templates per use case without rewriting everything.

// GET THIS SKILL — FREE