How Data Engineers Build Production RAG Pipelines
For Data engineers · Based on Grigorev RAG Application Build Framework
// TL;DR
Data engineers can use the Grigorev RAG framework to build a production-grade Retrieval Augmented Generation pipeline with clean separation between ingestion and query serving, persistent storage, and modular swappable components. The framework maps directly onto familiar engineering concerns: an ingestion process (fetch, clean, index), a persistent knowledge base (SQLite, Postgres, Elasticsearch), and a query-serving RAG assistant connected only through the shared database. Use it when you need to operationalise LLM-powered search over private data with maintainable, testable code and predictable cost.
Why is RAG a data engineering problem, not just an ML problem?
Most of a production RAG system is data infrastructure, not model tuning. The Grigorev framework makes this explicit: you treat the LLM as a black box and focus your effort on retrieval quality, data pipelines, and clean architecture. The three-step flow — Search, Build Prompt, LLM — is trivial to code; the real engineering is in ingestion, indexing, and process separation.
That's why this framework fits data engineers naturally. You already know how to build reliable ingestion jobs and manage persistent stores. RAG just adds a retrieval-and-generation layer on top.
How do you separate ingestion from the RAG assistant?
Run two independent processes connected only through a shared persistent database:
1. Ingestion process — fetches data from source, cleans it, structures each document into fields (`question`, `answer`, `section`, and keyword fields like `department`), and indexes it. Run it once or on a schedule.
2. RAG assistant — connects to the same database, performs search and generation on user queries.
Never mix these in one process. If you index on every app start (a common anti-pattern with in-memory tools like minsearch), you waste compute and lose data on restart. Persistent storage — SQLite FTS5, Postgres, or Elasticsearch — keeps the index on disk so indexing happens on your schedule, not per request.
How do you architect the code for swappable backends?
Extract logic into `ingest.py` (load_data, build_index) and a `rag_helper.py` housing a `RAGBase` class. Inject dependencies — `index` and `llm_client` — through `__init__` rather than using global variables. Each RAG component (`search`, `build_context`, `build_prompt`, `llm`, `rag`) is a method you can override.
To swap Elasticsearch for a vector DB, or OpenAI for Groq, subclass `RAGBase` and override only the affected method. Example: `LlamaRAG(RAGBase)` overriding just `llm()`. This is standard dependency injection — it makes your pipeline testable and provider-agnostic, and it prevents the tight coupling that plagues quick-prototype RAG code.
How do you tune retrieval for accuracy?
Distinguish text fields (question, answer, section — ranked full-text search) from keyword fields (department, category — exact-match filters that work like a SQL WHERE clause). Use keyword filters to exclude irrelevant sub-domains before ranking, then apply boosting to weight important fields — e.g. `boost=2` on the question field so query-to-question matches outrank answer-body matches. Return the top N (e.g. 5) results.
The rule: ensure the correct answer is somewhere in what you retrieve. The LLM decides what's relevant within that context, but it can't answer from documents you never fetched.
What are the production pitfalls to engineer around?
Avoid in-memory search in production, avoid mixing ingestion and serving in one process, and never use global variables for index and client — they block backend swaps. Track input/output token usage per query as a first-class metric for cost monitoring. Add guard rails against prompt injection on inputs and outputs, especially if the knowledge base contains sensitive data. And budget heavily for data cleaning — it's routinely the hardest, most underestimated step.
Next step: Scaffold `ingest.py` and `rag_helper.py` with a `RAGBase` class, stand up a SQLite FTS5 index, and run ingestion and the assistant as two separate processes against it. Then add token-usage logging and input/output guard rails.
// FREQUENTLY ASKED QUESTIONS
Which persistent store should I choose for a production RAG index?
For lightweight persistence, use SQLite FTS5 (wrapped by Grigorev's SQLite search) — same interface as minsearch but data survives restarts. For scale and advanced full-text features, use Elasticsearch. Postgres is a solid middle ground. Because the framework isolates the index behind an injectable dependency, you can migrate between them by changing your index setup, not your core RAG logic.
How do I make the RAG pipeline testable?
Inject the index and llm_client as dependencies via RAGBase's __init__ instead of using globals. This lets you pass mock or fixture backends in tests. Because search, build_prompt, and llm are separate methods, you can unit-test retrieval independently from generation — verifying the correct docs are retrieved before ever calling the LLM.
How often should the ingestion process run?
Run ingestion on a schedule that matches how often your source data changes — nightly is common for stable knowledge bases. Because ingestion is a separate process connected only through the persistent database, you can re-index without touching the live RAG assistant. For frequently changing data, trigger incremental ingestion on document updates.
How do I monitor a RAG pipeline in production?
Track input and output token usage per query as a core metric — it drives cost and flags over-retrieval. Log retrieval hit quality (whether the answer was in the top N), ingestion run success, and index freshness. Extract token counts from the LLM API response inside your llm() method so monitoring is built into the pipeline, not bolted on.