How to Ship an Accurate RAG Feature Without Building From Scratch
For Technical founders building AI products · Based on Cole Medin RAG 2.0 Agentic Knowledge System
// TL;DR
Technical founders building AI products use Cole Medin's RAG 2.0 template to ship a knowledge retrieval feature that handles both factual and relational queries without months of research. The free template combines pgvector and a Neo4j knowledge graph under an agent that routes queries via a customizable system prompt. Use it when your product's users ask questions naive RAG can't answer — comparisons, entity relationships, multi-hop lookups — and you need something more accurate than a similarity-search MVP before your next demo or funding milestone.
Why won't a simple vector-search MVP hold up?
Most founders start with naive RAG: chunk docs, embed, retrieve, inject. It demos well on softball factual questions and falls apart the moment a customer asks something relational — 'Which of these companies partnered with each other?' or 'Compare product A and product B across integrations.' Vector similarity retrieves chunks that sound related but can't traverse actual connections between entities. Your churn risk isn't the model; it's the retrieval architecture.
RAG 2.0 fixes this with Dual Representation and an agent that routes. You store the same corpus in a vector database for factual lookups and a knowledge graph for relational traversal, then let the agent decide per query. It's the difference between an impressive demo and a product users trust.
How fast can you actually ship this?
Cole Medin's template is free and does the heavy lifting. Spin up Neon's free-tier Postgres with pgvector, deploy Neo4j locally, copy .env.example to .env, and fill in your providers. Crucially, your main LLM and embedding provider can differ — so you can use OpenRouter for reasoning and OpenAI for embeddings. Use a cheap ingestion LLM like GPT-4.1 Nano so your graph-building costs stay negligible even as your document set grows.
Drop your markdown docs in /documents, run `python -m ingestion.ingest --clean`, and you have a working dual-store system in an afternoon. If your product only needs factual lookups for now, pass the no-knowledge-graph flag and ship vector-only — then add the graph when relational queries become a real user need.
Where should you spend your engineering effort?
Don't over-invest in infrastructure — the template handles it. Spend your time on agent/prompts.py, the routing logic. This is the 10% of domain-specific work that determines product quality. Name your entities and relationships explicitly and tell the agent when to use vector search, graph search, or both. Founders who vibe-code this step and ship the generic template get inconsistent, low-quality answers.
Validate with the CLI, which shows tool calls live. Test single-entity, relational, and combined queries. Remember LLMs are non-deterministic — judge answer quality, not exact routing. Iterate on the prompt, not the plumbing.
How do you frame this for investors and customers?
RAG 2.0 gives you a defensible technical story: 'Our agent reasons about how to retrieve knowledge and traverses entity relationships competitors' vector-only systems can't.' That's a concrete accuracy advantage you can demo. Show a side-by-side where naive RAG whiffs on a relational question and your agent nails it by routing to the knowledge graph.
For cost, emphasize the asymmetry: expensive work happens once at ingestion (which you can batch and control with a lightweight LLM), while runtime queries stay fast and cheap. That's a healthy unit-economics story for a knowledge product.
Next step: Clone the RAG 2.0 template, ingest a slice of your real product data, and run three relational queries through the CLI — if the agent answers them correctly where your current MVP fails, you've found your differentiator.
// FREQUENTLY ASKED QUESTIONS
Do I need a knowledge graph at MVP stage or can I add it later?
Start vector-only if your users mostly ask factual lookups — pass the no-knowledge-graph flag for fast, cheap ingestion. Add the knowledge graph when relational or comparison queries become a real user need. The template supports both, so you can layer in graph capability without re-architecting, only re-ingesting to populate the graph via Graffiti.
How much will the LLM costs run for a small product?
Costs are front-loaded at ingestion, where you should use a lightweight model like GPT-4.1 Nano to keep entity extraction cheap. Runtime queries use your main LLM but are infrequent per user. For a small product with a modest corpus, this stays inexpensive — and you can use Ollama locally to eliminate API costs entirely if privacy or budget demands it.
Can I demo RAG 2.0's advantage over competitors easily?
Yes — run a side-by-side where a naive vector-only setup fails a relational question ('How are X and Y connected?') and your agent answers correctly by routing to the knowledge graph. The CLI shows the tool invocation live, giving you a concrete, visual proof point of the agentic routing advantage for investor and customer demos.