Cole Medin RAG 2.0 Agentic Knowledge System

Last updated: 3 July 2026

Build a knowledge retrieval system where an AI agent reasons about whether to query a vector database, a knowledge graph, or both — delivering far more accurate and relational answers than naive RAG alone.

// TL;DR

Cole Medin's RAG 2.0 Agentic Knowledge System is a retrieval architecture that lets an AI agent decide — at query time — whether to search a vector database, a Neo4j knowledge graph, or both, based on the nature of each question. Store the same documents twice (chunked in pgvector for factual lookups, as entities and relationships in a knowledge graph for relational queries) and let the agent's system prompt route each query. Use it whenever your knowledge base contains both discrete factual lookups AND relational/entity connections that naive similarity search alone cannot surface accurately.

Framework

// When should you use the RAG 2.0 Agentic Knowledge System?

Use this skill whenever you need to design or implement an AI agent's knowledge retrieval layer, especially when your data contains both discrete factual lookups AND relational/entity connections that benefit from different retrieval strategies.

// What do you need before building an agentic knowledge system?

knowledge_domainrequired
The subject matter or document corpus the agent will search (e.g. company AI initiatives, product documentation, research papers)
query_typesrequired
Examples of the kinds of questions users will ask — single-entity lookups vs. relational/comparison questions
llm_providerrequired
The LLM provider to use (OpenAI, Ollama, Gemini, OpenRouter, or any OpenAI-API-compatible provider)
embedding_modelrequired
The embedding model for vector storage (e.g. text-embedding-3-small from OpenAI, or an Ollama model); note the vector dimensions
existing_documentsrequired
Markdown documents or other source material to ingest into the knowledge base
relational_entities
The key entities and relationships in your domain that make knowledge graph traversal worthwhile (e.g. companies, partnerships, investments)

// What core principles make RAG 2.0 work?

Vanilla RAG / Naive RAG Problem

Traditional RAG is a fixed, inflexible pipeline: embed the query, retrieve the closest chunks, inject them into the prompt. The agent cannot refine its search, choose a different source, or reason about how to explore the knowledge. It is force-fed context whether the context is appropriate or not.

Agentic RAG

Agentic RAG gives the agent the ability to reason about how it explores the knowledge base instead of always force-feeding context as a pre-processing step. The agent picks tools — vector search, graph search, web search, or combinations — based on the nature of the user's question.

Dual Representation

Store the same information in two ways simultaneously: chunked and embedded in a vector database for discrete factual lookups, and as entities and relationships in a knowledge graph for relational queries. Neither representation replaces the other — they are complementary.

Agent-Controlled Routing

The agent, not the pipeline, decides which retrieval tool to invoke. This routing logic is defined in the system prompt. The system prompt must be customised to the specific knowledge domain so the agent knows when each strategy is appropriate.

Computational Cost Asymmetry

Ingesting into a vector database takes seconds; ingesting into a knowledge graph takes minutes because LLMs must define all entities and relationships. However, querying the knowledge graph at runtime is fast. Budget extra time at ingestion, not at query time.

Lightweight Ingestion LLM

Use a smaller, cheaper model (e.g. GPT-4.1 Nano) for the ingestion process that transforms documents into knowledge graph entities and relationships. Reserve your main LLM for the agent's reasoning and response generation.

// How do you build the RAG 2.0 system step by step?

1
Classify your knowledge domain by query type
Before writing any code, map the questions users will ask into two buckets: (A) single-entity lookups best served by a vector search (e.g. 'What are Google's AI initiatives?') and (B) relational/comparison queries best served by a knowledge graph (e.g. 'How are OpenAI and Microsoft related?'). If both types exist, you need both retrieval tools.
2
Set up your Postgres + pgvector instance as the vector database
Use a Postgres platform (Neon recommended for free tier and pgvector support). Run the SQL schema to create your chunks and documents tables. If you use a non-default embedding model, update the vector dimension value (default: 1536 for text-embedding-3-small) in every place it appears in the schema. Run the schema in a fresh project to avoid destructive drops on existing data.
3
Set up Neo4j as the knowledge graph engine
Deploy Neo4j either via a local AI package (e.g. the Cole Medin Local AI Package) or Neo4j Desktop. Capture the bolt connection URL (default: bolt://localhost:7687), username, and password for environment configuration.
4
Configure your environment variables
Copy .env.example to .env. Set: DATABASE_URL (Postgres connection string), Neo4j bolt URL + credentials, LLM provider + base URL + API key + model name, embedding provider + base URL + API key + embedding model, and a separate (lighter) ingestion LLM. The LLM provider and embedding provider can differ — useful when your main provider (e.g. OpenRouter) does not offer embeddings.
5
Place markdown documents into the /documents folder and run ingestion
Run `python -m ingestion.ingest --clean` (the --clean flag wipes existing data for a fresh start). The ingestion script: (a) chunks and embeds documents into Postgres/pgvector in seconds, and (b) uses the ingestion LLM to extract entities and relationships and writes them into Neo4j via Graffiti — this takes minutes. Monitor Graffiti 'episodes' appearing in the Neo4j dashboard. Do not interrupt. If you need speed without relational queries, pass the no-knowledge-graph flag.
6
Author the agent system prompt to encode routing logic
Edit agent/prompts.py. This is the most domain-specific step. Define explicit rules for when the agent uses: (a) vector store tool only — single-entity lookups, (b) knowledge graph tool only — relational/comparison questions involving two or more entities, (c) both tools — complex queries requiring breadth and relational depth. Be specific about the entity types and relationship types present in your knowledge graph. Do not leave this as the generic template; tailor it to your data.
7
Start the FastAPI agent endpoint and CLI
Terminal 1: run `python -m agent.api`. Wait for 'graph connection successful'. Terminal 2: run `python cli.py`. The CLI shows which tools the agent invoked in real time — use this to validate your routing logic. Test with at least one single-entity query, one relational query, and one that should trigger both tools.
8
Validate routing behaviour and iterate on the system prompt
Because LLMs are non-deterministic, routing will not be identical every run. Judge on answer quality, not on exact tool selection. If the agent consistently routes incorrectly (e.g. always uses vector search for relational questions), refine the system prompt routing rules in prompts.py. Do not change the underlying infrastructure — the system prompt is the lever.

// What does RAG 2.0 look like in real knowledge domains?

A knowledge base about enterprise software vendors and their partnership ecosystems

Single-entity query ('What features does Vendor X offer?') routes to vector search. Relational query ('How are Vendor X and Vendor Y integrated?') routes to the knowledge graph, where entities like vendors, products, and integration types are nodes and 'integrates_with' or 'acquired_by' are relationship edges. The agent system prompt is written to trigger graph search whenever two vendor names appear in the same question.

A medical research knowledge base covering clinical trials and institutions

Factual lookups ('What are the side effects of Drug A?') use the vector database over chunked trial documents. Relational queries ('Which institutions collaborated on Drug A and Drug B trials?') use the knowledge graph where institutions, drugs, and trials are nodes with 'conducted_by' and 'studied_in' relationships. The ingestion LLM extracts these entities automatically during the Graffiti ingestion step.

// What mistakes should you avoid when building agentic RAG?

Using Naive/Vanilla RAG for knowledge that is inherently relational — vector similarity alone cannot surface entity relationships.
Leaving the agent system prompt as the generic template — routing behaviour is only as good as the domain-specific rules you encode in prompts.py.
Forgetting to update vector dimensions in the SQL schema when switching embedding models — mismatched dimensions will cause silent failures or errors.
Running the --clean ingestion flag on a production database — it destroys and recreates all tables.
Expecting knowledge graph ingestion to be fast — it takes minutes per document because LLMs must extract all entities and relationships; interrupting the process leaves an incomplete graph.
Assuming the agent will route identically every run — LLMs are non-deterministic; evaluate on answer quality, not on exact tool call sequence.
Using a heavyweight LLM for ingestion when a lightweight model (e.g. GPT-4.1 Nano) is sufficient — this increases cost significantly for large corpora.
Vibe coding the agent without understanding the architecture — always validate LLM-generated code and add the final 10% of domain-specific logic yourself.

// What are the key terms in the RAG 2.0 system?

Vanilla RAG / Naive RAG: The simplest RAG pipeline: chunk documents, embed them, store in a vector database, embed the user query, retrieve the most similar chunks, inject as context. Inflexible — the agent cannot choose how or whether to retrieve.
Agentic RAG: A retrieval approach where the agent reasons about how to explore the knowledge base, selecting from multiple retrieval tools (vector search, graph search, web search) based on the nature of the question, rather than having context force-fed as a pre-processing step.
Knowledge Graph: A relational representation of information as nodes (entities) and edges (relationships), stored in Neo4j. Enables traversal of connections between entities that vector similarity search cannot express.
Dual Representation: Storing the same document corpus in both a vector database (for semantic/factual lookup) and a knowledge graph (for relational/entity traversal) simultaneously, giving the agent two fundamentally different lenses on the same data.
Graffiti: The Python knowledge graph library used alongside Neo4j to extract entities and relationships from documents using LLMs and write them as 'episodes' into the graph store.
pgvector: A Postgres extension that adds vector storage and similarity search to a standard SQL database, enabling Postgres to act as a vector database without a separate specialised store.
Ingestion LLM: A separate, lightweight LLM (distinct from the main agent LLM) used specifically during document ingestion to extract entities and relationships for the knowledge graph. Chosen for cost-efficiency, not capability.
Agent Routing Logic: The rules encoded in the agent's system prompt (prompts.py) that instruct the agent when to use vector search, knowledge graph search, or both. Must be customised per knowledge domain.
RAG 2.0: Cole Medin's term for the combination of Agentic RAG + Knowledge Graphs — representing the evolution from static, pipeline-driven retrieval to agent-controlled, multi-strategy knowledge exploration.
Claude.md: The global rules file for the Claude Code AI coding assistant — analogous to cursor rules or windsurf rules — defining how the assistant should use planning files, MCP servers, and unit testing throughout a build session.
Planning Mode: A Claude Code mode (activated by pressing Shift+Tab twice) that forces the assistant to plan without writing to the filesystem, used to generate planning.md and task.md before any code is written.
planning.md / task.md: The two artefacts produced during Claude Code's planning mode: planning.md describes the high-level architecture and tech stack; task.md is the ordered checklist of build tasks the assistant will execute sequentially.

// FREQUENTLY ASKED QUESTIONS

What is Agentic RAG?

Agentic RAG is a retrieval approach where the AI agent reasons about how to explore the knowledge base — selecting from multiple tools like vector search, graph search, or web search based on the question — instead of having context force-fed as a fixed pre-processing step. The agent, not the pipeline, decides how and whether to retrieve.

What is RAG 2.0 in Cole Medin's system?

RAG 2.0 is Cole Medin's term for combining Agentic RAG with knowledge graphs. It stores documents in both a vector database (pgvector) for factual lookups and a Neo4j knowledge graph for relational queries, then lets an AI agent choose which to query per question — representing the evolution from static, pipeline-driven retrieval to agent-controlled, multi-strategy exploration.

How do I build an agentic RAG system with a knowledge graph?

Classify your queries into single-entity lookups versus relational questions, set up Postgres with pgvector and Neo4j, configure environment variables including a lightweight ingestion LLM, ingest documents into both stores, then author a domain-specific system prompt in prompts.py that encodes routing logic. Start the FastAPI endpoint and CLI, then validate the agent's tool selection on real queries.

How do I decide when to use a vector database versus a knowledge graph?

Use a vector database for single-entity factual lookups (e.g. 'What features does Vendor X offer?') and a knowledge graph for relational or comparison queries involving two or more entities (e.g. 'How are OpenAI and Microsoft related?'). If your domain contains both query types, store data in both and let the agent route each question accordingly.

How does RAG 2.0 compare to naive RAG?

Naive RAG is a fixed pipeline: embed the query, retrieve the closest chunks, inject them into the prompt — the agent can't refine its search or choose sources. RAG 2.0 gives the agent multiple retrieval tools and lets it reason about which to use, adding a knowledge graph for relational queries that vector similarity alone cannot surface.

When should I use a knowledge graph instead of just vector search?

Use a knowledge graph whenever your data is inherently relational — when questions ask how entities connect, compare, or influence each other. Vector similarity search retrieves semantically similar chunks but cannot express or traverse relationships between entities. If users ask 'how are X and Y related?', vector search alone will fail and a knowledge graph is essential.

What results can I expect from an agentic knowledge system?

Expect far more accurate and relational answers than naive RAG delivers, especially on comparison and multi-entity questions. The agent routes factual lookups to vector search and relational queries to the graph, avoiding the force-fed, often-irrelevant context of fixed pipelines. Answer quality improves as you refine domain-specific routing rules in the system prompt.

What is Graffiti in this RAG setup?

Graffiti is the Python knowledge graph library used alongside Neo4j to extract entities and relationships from documents using an LLM and write them as 'episodes' into the graph store. During ingestion, Graffiti processes each document — a slow, minutes-per-document step — populating the graph the agent later queries quickly at runtime.

Why does knowledge graph ingestion take so long?

Knowledge graph ingestion takes minutes per document because the LLM must read each document and extract all entities and relationships, which is computationally heavy. Vector ingestion, by contrast, takes seconds. This cost asymmetry is front-loaded at ingestion — querying the graph at runtime is fast, so budget extra time for ingestion, not queries.

Can I use different providers for my LLM and embeddings?

Yes — the LLM provider and embedding provider can differ, which is useful when your main provider (e.g. OpenRouter) does not offer embeddings. Configure DATABASE_URL, Neo4j credentials, a main LLM provider, a separate embedding provider, and a lightweight ingestion LLM independently in your .env file.

// GET THIS SKILL — FREE