How PMs Scope a RAG Feature That Actually Ships
For Technical product managers · Based on Grigorev RAG Application Build Framework
// TL;DR
Technical product managers can use the Grigorev RAG framework to scope, estimate, and de-risk a Retrieval Augmented Generation feature — an LLM that answers from your product's private data without model retraining. The framework's three-step flow (Search, Build Prompt, LLM) gives PMs a shared vocabulary with engineers, while its principles surface the real cost and effort drivers: data preparation, retrieval quality, persistent storage, and process separation. Use it when planning an AI answers feature so your estimates, milestones, and risk register reflect where the actual work lives.
What is RAG and why does it matter for your roadmap?
RAG (Retrieval Augmented Generation) lets an LLM answer questions from your private data — docs, FAQs, product records — without the cost or delay of retraining a model. For a PM, this is the fastest path to an "AI answers" feature grounded in your own content. The Grigorev framework breaks every RAG system into three steps you can put on a roadmap: Search (find relevant docs), Build Prompt (assemble context and question), and LLM (generate the answer).
Knowing this three-step flow lets you talk precisely with engineers and avoid scoping the feature as a vague "add AI" ticket.
How should you estimate the effort for a RAG feature?
The biggest surprise for PMs: data preparation dominates the timeline. Getting your content into clean, machine-readable documents with the right fields (question, answer, section, and a keyword field like category) is routinely the hardest, most underestimated step. The three-function code (search, build_prompt, llm) is fast; cleaning and structuring data is not.
Scope in two phases:
- Phase 1 — Prototype: in-memory search (minsearch), pre-cleaned subset of data, validate answer quality. Days, not weeks.
- Phase 2 — Production: persistent storage, separated ingestion pipeline, guard rails, monitoring. This is where most of the calendar goes.
Set stakeholder expectations that a working demo is quick but production hardening is the real investment.
What are the cost and risk drivers you need on the register?
Cost is driven by token usage — how much retrieved context plus the question you send per query. If engineers send all documents instead of a retrieved subset, costs balloon and answer quality drops. Insist on retrieval (top N results) and per-query token tracking as acceptance criteria.
Key risks to log:
- Prompt injection — users crafting inputs like "ignore all previous instructions." Require guard rails on inputs and outputs before any public launch, especially with sensitive or NDA data.
- Data staleness — ensure ingestion runs on a schedule so answers stay current.
- Vendor lock-in — ask for modular, swappable architecture (a RAGBase class with injectable dependencies) so switching LLM providers is a one-method change, not a rebuild.
How do you know the feature is working?
Define acceptance around retrieval and grounding, not just "it responds." The LLM should answer only from provided context and reply "I don't know" when the answer isn't retrieved — that's a testable behavior. Validate that correct answers appear in the top N retrieved results, since the LLM can't answer from documents that were never fetched. Track token usage per query against a cost target.
These give you objective, demoable success criteria rather than subjective "feels smart" judgments.
What should you push engineering to build for the long term?
Request separation of the ingestion process from the RAG assistant, persistent storage over in-memory search, and modular code so backends and providers stay swappable. These aren't gold-plating — they're what keeps the feature maintainable, cost-controllable, and future-proof as pricing and models change.
Next step: Write a one-page brief mapping your feature to the three-step RAG flow, flag data preparation as the critical path, and add prompt-injection guard rails plus token tracking to your acceptance criteria before kickoff.
// FREQUENTLY ASKED QUESTIONS
How long does it take to ship a RAG feature?
A working prototype with minsearch and pre-cleaned data can take days. Production readiness — persistent storage, separated ingestion, guard rails, and monitoring — takes considerably longer, largely because data preparation is the most underestimated step. Scope it in two phases and set expectations that the demo is fast but hardening the pipeline is the real investment.
Do we need to retrain or fine-tune an LLM for this?
No. RAG answers from your private data by retrieving relevant documents and injecting them into the prompt at query time — no retraining. This is faster to update (just re-index new content) and cheaper than fine-tuning. It's why RAG is the most common LLM application pattern in industry for domain-specific question answering.
What's the main risk to flag for a public-facing RAG feature?
Prompt injection — users crafting inputs like 'ignore all previous instructions' to override your system prompt. Require guard rails on both inputs and outputs before launch, and never expose a system containing sensitive or NDA data publicly without them. Add data staleness and vendor lock-in to the risk register too.
How do we avoid getting locked into one LLM provider?
Ask engineering to use modular, swappable architecture — a RAGBase class with injectable index and llm_client dependencies. Switching providers (OpenAI to Anthropic, Gemini, or Groq) then means overriding one method, not rewriting the system. The framework is deliberately provider-agnostic, so make swappability an explicit requirement in your brief.