Frequently Asked Questions About Cole Medin RAG 2.0 Agentic Knowledge System
20 answers covering everything from basics to advanced usage.
// Basics
What is Dual Representation in RAG 2.0?
Dual Representation means storing the same document corpus in two ways simultaneously: chunked and embedded in a vector database for semantic factual lookups, and as entities and relationships in a knowledge graph for relational traversal. Neither replaces the other — they're complementary lenses on the same data, giving the agent two fundamentally different ways to explore knowledge.
What is an ingestion LLM and why is it separate?
The ingestion LLM is a lightweight model (e.g. GPT-4.1 Nano) used only during document ingestion to extract entities and relationships for the knowledge graph. It's kept separate from the main agent LLM because extraction is a high-volume, repetitive task where cost matters more than reasoning quality — using a heavyweight model here inflates costs for large corpora.
What is pgvector and why use Postgres as a vector database?
pgvector is a Postgres extension that adds vector storage and similarity search to a standard SQL database, letting Postgres act as a vector database without a separate specialised store. This simplifies your stack — you can use one database (Neon's free tier is recommended) for both structured data and embeddings rather than maintaining a dedicated vector store.
What free-tier tools can I use to build this without paying?
Use Neon's free tier for Postgres with pgvector, a locally deployed Neo4j (via Neo4j Desktop or a local AI package), Ollama for local LLM and embeddings, or free-tier API credits from OpenAI/Gemini/OpenRouter. The main cost driver is knowledge graph ingestion, which you can minimize by using a lightweight ingestion LLM like GPT-4.1 Nano.
// How To
How do I set up Neo4j for the knowledge graph?
Deploy Neo4j either via a local AI package (like the Cole Medin Local AI Package) or Neo4j Desktop. Capture the bolt connection URL (default bolt://localhost:7687), username, and password, then add them to your .env file. During ingestion you can monitor Graffiti 'episodes' appearing in the Neo4j dashboard to confirm the graph is populating.
How do I write the agent's routing logic?
Edit agent/prompts.py — the most domain-specific step. Define explicit rules for when the agent uses vector search only (single-entity lookups), knowledge graph only (relational or comparison questions involving two or more entities), or both (complex queries needing breadth and relational depth). Be specific about the entity types and relationship types in your graph. Never leave it as the generic template.
How do I run ingestion for a fresh knowledge base?
Place your markdown documents in the /documents folder, then run `python -m ingestion.ingest --clean`. The --clean flag wipes existing data for a fresh start. The script chunks and embeds into pgvector in seconds, then uses the ingestion LLM to extract entities and relationships into Neo4j via Graffiti over several minutes. Don't interrupt it.
How do I test whether my routing logic works?
Run the FastAPI agent endpoint (`python -m agent.api`, wait for 'graph connection successful') and the CLI (`python cli.py`) in separate terminals. The CLI shows which tools the agent invokes in real time. Test at least one single-entity query, one relational query, and one that should trigger both tools to validate your routing.
// Troubleshooting
My agent always uses vector search for relational questions. How do I fix it?
Refine the routing rules in prompts.py — do not change the underlying infrastructure. The system prompt is the lever. Make the rule for graph search more explicit, for example: 'Whenever a question mentions two or more entities by name, or asks how things relate, compare, or connect, use the knowledge graph tool.' Then re-test on the same queries.
Why is my ingestion failing with vector dimension errors?
You likely switched embedding models without updating the vector dimension in the SQL schema. The default is 1536 for text-embedding-3-small — if your model produces different dimensions, update the value in every place it appears in the schema. Mismatched dimensions cause silent failures or errors, so verify your embedding model's output dimensions and align the schema.
Why does the agent route differently on identical questions?
LLMs are non-deterministic, so tool selection won't be identical every run. This is expected behaviour. Judge the system on answer quality, not on exact tool call sequences. Only intervene if the agent consistently routes incorrectly — for example, always using vector search for clearly relational questions — in which case refine the prompt rules.
I accidentally interrupted knowledge graph ingestion. What happens?
Interrupting leaves an incomplete graph — some documents' entities and relationships won't have been extracted, so relational queries over those documents will return partial or wrong results. Re-run ingestion, ideally with --clean on a non-production database, and let it complete. Monitor Graffiti episodes in the Neo4j dashboard to confirm all documents processed.
// Comparisons
How does RAG 2.0 compare to using a vector database alone?
A vector database alone handles semantic and factual lookups well but cannot express or traverse relationships between entities — so 'how are X and Y related?' fails. RAG 2.0 adds a knowledge graph for exactly those relational queries and an agent to route each question to the right store, giving accurate answers across both factual and relational question types.
How does Agentic RAG differ from GraphRAG?
GraphRAG focuses on building and querying a knowledge graph for retrieval. RAG 2.0's Agentic RAG is broader: it keeps both a vector database and a knowledge graph, then adds an agent that reasons about which tool — vector, graph, or both — fits each question. It's less about a single graph technique and more about agent-controlled multi-strategy routing.
Is a knowledge graph worth the extra ingestion cost?
It's worth it when your data is genuinely relational and users ask comparison or multi-entity questions. If your queries are purely single-entity factual lookups, the graph adds ingestion time and cost with little benefit — you can pass the no-knowledge-graph flag. Classify your query types first; only build the graph if relational queries genuinely exist.
// Advanced
Can I use Ollama instead of OpenAI for a fully local system?
Yes — the system supports Ollama and any OpenAI-API-compatible provider for both the LLM and embedding model. Configure the provider, base URL, API key, and model name in .env. Pair Ollama with a locally deployed Neo4j and self-hosted Postgres for a fully local, private RAG 2.0 stack, though local models may extract graph entities less reliably.
How should I structure entities and relationships for my domain?
Identify the key entity types (e.g. companies, products, institutions, drugs) and the relationship edges connecting them (e.g. 'integrates_with', 'acquired_by', 'conducted_by'). The ingestion LLM extracts these automatically via Graffiti, but you must describe them explicitly in prompts.py so the agent knows what the graph contains and when graph traversal will answer a question.
Can the agent use web search alongside vector and graph search?
Yes — Agentic RAG's core idea is that the agent selects from multiple retrieval tools, which can include web search, vector search, and graph search or any combination. You'd add the web search tool and encode routing rules in the system prompt describing when to reach beyond the internal knowledge base — for example, for recent or out-of-corpus information.
How do I speed up ingestion when I don't need relational queries?
Pass the no-knowledge-graph flag during ingestion. This skips the slow LLM-based entity and relationship extraction into Neo4j and only chunks and embeds documents into pgvector — which takes seconds. Use this when your domain is purely factual lookups and you want a fast, vector-only setup without the knowledge graph overhead.
Should I vibe-code the agent with Claude Code or build it manually?
You can scaffold with Claude Code using planning mode to produce planning.md and task.md, but always validate the generated code and add the final 10% of domain-specific logic yourself — especially the routing rules in prompts.py. Vibe coding without understanding the dual-store architecture leads to fragile systems that route poorly and fail silently.