Frequently Asked Questions About Grigorev RAG Application Build Framework
22 answers covering everything from basics to advanced usage.
// Basics
What does the R in RAG actually mean?
The R stands for Retrieval, which is simply search. You query your knowledge base with the user's question to fetch the top N documents most likely to contain the answer, then pass only those to the LLM. Retrieval and search are synonyms in RAG — there's no hidden complexity. It's cheaper, faster, and more accurate than sending all your data every time.
Do I need to understand how the LLM works internally to build RAG?
No. Treat the LLM as a black box — you only need to know how to feed it the right inputs and read its outputs. The skill of RAG engineering lies entirely in what you put into the LLM (prompt and retrieval design), not how the LLM works internally. This lets you focus on retrieval quality and prompt structure rather than model architecture.
What is a knowledge base in the context of RAG?
A knowledge base is the searchable repository of documents your RAG system retrieves from — FAQs, policies, product records, or documentation. It must be indexed for efficient search. Each document should have fields like question, answer, section, and a keyword field (course, department, category) for filtering. It's the private data source that lets the LLM answer questions it was never trained on.
What is the difference between ingestion and the RAG assistant?
Ingestion is the process of fetching data from its source, cleaning it, and indexing it into the knowledge base. The RAG assistant is the process that queries that knowledge base and generates answers to user questions. In production, these run as separate processes connected only through a shared persistent database, so indexing happens once or on a schedule — not on every app start.
// How To
How do I implement the search function in a RAG system?
Write a search(question) function that sends the user query to the index, applies a keyword filter to restrict results to the relevant sub-domain, applies boost weights to prioritise important fields (e.g. question field gets boost=2), and returns the top N results (e.g. top 5). This is the R in RAG — the retrieval step that feeds relevant context into the prompt.
How do I write the build_prompt function?
Create a build_prompt(question, search_results) function with two components: Instructions (a constant system prompt telling the LLM its task) and a User Prompt Template (a string that formats the question and retrieved context). First convert each retrieved document dict into a readable string using a build_context helper, then inject that context and the question into the template. Keep instructions separate from the template in code.
How do I track token usage and costs in RAG?
In your llm() function, extract input and output token counts from the API response and log them per query. Monitoring input/output tokens lets you estimate costs and optimise for efficiency. Since retrieval controls how much context you send, tracking tokens also tells you if you're retrieving too many documents. Without this visibility you cannot forecast or control your RAG system's operating cost.
How do I make my RAG system swappable between search backends and LLM providers?
Use modular code architecture: write search, build_prompt, and llm as separate methods in a RAGBase class with injectable dependencies (index, llm_client) passed via __init__ — never global variables. To swap components, subclass and override only the relevant method. For example, create LlamaRAG(RAGBase) and override just the llm() method to switch providers, leaving retrieval logic untouched.
How do I move a RAG prototype into production?
Switch from in-memory search (minsearch) to persistent storage (SQLite, Postgres, Elasticsearch), then split into two independent processes: an ingestion process that fetches and indexes data on a schedule, and a RAG assistant that connects to the same database to answer queries. Add guard rails against prompt injection on inputs and outputs, and monitor token usage to track costs.
// Troubleshooting
Why is my RAG system returning irrelevant or wrong answers?
Usually your retrieval step isn't surfacing the right documents. Check that the answer actually exists in your top N results — the LLM can only work from what you retrieve. Apply keyword filters to exclude irrelevant sub-domains, use boosting to prioritise the question field, and verify your data is clean and properly indexed. If you're sending all documents instead of a relevant subset, irrelevant content confuses the LLM.
Why does my RAG app lose all its data when I restart it?
You're likely using in-memory search like minsearch, which stores the index in memory and loses everything when the process stops. This forces a full re-ingestion on every restart. For anything beyond a quick prototype, switch to persistent storage (SQLite, Postgres, Elasticsearch) that keeps data on disk across sessions, and separate ingestion from the assistant so indexing happens once, not on every startup.
Why is my RAG application so expensive to run?
You're probably sending too much context to the LLM — either all your documents instead of a small retrieved subset, or too many retrieved results per query. Retrieval exists to send only the documents likely to contain the answer, cutting token costs dramatically. Track input/output tokens per query, tune your top N down, and use keyword filters to narrow the search space before ranking.
Why is it so hard to get my RAG system working with real data?
Data preparation is the most underestimated step. Getting real-world data into clean, machine-readable format is routinely the hardest and most time-consuming part of a RAG project. The framework's examples use pre-cleaned data, so expect significant effort in production. LLMs can assist with cleaning, but plan for meaningful time on structuring documents into fields like question, answer, section, and keyword fields.
// Comparisons
How does RAG compare to sending all your documents to the LLM's context window?
RAG retrieves only a small relevant subset while dumping everything relies on a large context window. RAG is cheaper (fewer tokens), faster, and more accurate — large context stuffing confuses the LLM with irrelevant content and degrades answer quality. Even with large context windows, retrieval remains the better pattern because it focuses the model on what matters and keeps costs predictable.
How does minsearch compare to Elasticsearch for RAG?
minsearch is in-memory, dependency-free, and ideal for prototypes but loses all data on restart. Elasticsearch is persistent, scalable, and production-ready with advanced full-text search. Both share similar concepts (text fields, keyword fields, boosting). With modular architecture you can start on minsearch and swap to Elasticsearch by changing one function. Use minsearch to learn and prototype, Elasticsearch for scale and persistence.
How does the responses API compare to chat completions for RAG?
For OpenAI, the responses API is preferred over chat completions in this framework. Both send a message history with system/developer and user roles, but the responses API offers a cleaner interface for the RAG pattern. Whichever you use, keep instructions in the system/developer role and the user's question plus retrieved context in the user role, and extract the text response consistently.
How does keyword filtering compare to a vector database for narrowing results?
Keyword filtering does exact-match filtering (like a SQL WHERE clause) to restrict the search space before ranking — cheap and precise for structured fields like department or course. Vector databases rank by semantic similarity across all documents. They're complementary: use keyword fields to exclude irrelevant sub-domains entirely, then rank the remaining candidates. This framework emphasises keyword filtering plus full-text ranking as a strong, simple baseline.
// Advanced
What is boosting and how do I use it effectively?
Boosting assigns a numerical importance weight to a field during search. A field with boost=2 is twice as valuable as one with boost=1 when ranking results. Use it to prioritise matches where they matter most — for example, boost the question field over the answer field so documents whose question closely matches the user's query rank higher. Tune boost values by testing which retrievals surface correct answers.
How should I design the RAGBase class for maximum flexibility?
Give RAGBase methods for search(), build_context(), build_prompt(), llm(), and rag(), with dependencies (index, llm_client) injected via __init__ rather than hardcoded or global. This encapsulation means each component is swappable. To change providers or backends, subclass RAGBase and override only the method that differs — e.g. LlamaRAG(RAGBase) overriding llm() — leaving retrieval and prompt logic intact.
How do I schedule ingestion separately from the RAG assistant in production?
Run ingestion as an independent process that fetches source data, cleans it, and indexes it into a persistent database on a schedule (e.g. nightly). The RAG assistant runs as a separate process connecting to the same database to serve queries. They share nothing but the database, so re-indexing new documents never disrupts live query serving and indexing happens once per schedule, not per request.
What guard rails should I add for a public-facing RAG system?
Add input guard rails to detect prompt injection attempts like 'ignore all previous instructions,' and output guard rails to verify responses are appropriate before returning them. Never expose a RAG system containing NDA-protected or sensitive data publicly without both. Keeping instructions separate from the user prompt also helps contain injection, and instructing the LLM to answer only from provided context limits leakage of unrelated knowledge.
Can I use RAG with any LLM provider?
Yes — the framework is provider-agnostic. It works with OpenAI, Anthropic, Gemini, Groq, or others; OpenAI is just the default example. Because you treat the LLM as a black box and isolate the llm() method behind an injectable dependency, switching providers means changing only that one function. Format inputs and read outputs consistently, and the rest of your RAG pipeline stays unchanged.