How to Build an Agentic RAG Assistant for Enterprise Data
For AI engineers building enterprise knowledge assistants · Based on Cole Medin RAG 2.0 Agentic Knowledge System
// TL;DR
AI engineers building enterprise knowledge assistants use Cole Medin's RAG 2.0 to handle the reality that corporate data contains both factual lookups (product specs, policies) and relational questions (vendor integrations, org structures). By storing documents in both pgvector and a Neo4j knowledge graph and letting an agent route each query via a domain-tuned system prompt, you deliver accurate answers where naive RAG fails on relational queries. Use it when your assistant must answer both 'what does X do?' and 'how are X and Y connected?' over the same corpus.
Why does naive RAG break down on enterprise data?
Enterprise knowledge bases are rarely just flat facts. They describe vendors, products, teams, integrations, acquisitions, and dependencies — a web of relationships. Naive RAG embeds a query, retrieves the nearest chunks, and force-feeds them into the prompt. That works for 'What features does our billing platform offer?' but collapses on 'Which of our vendors integrate with both Salesforce and our data warehouse?' Vector similarity cannot express or traverse relationships between entities, so the assistant returns semantically adjacent chunks that miss the actual connection.
RAG 2.0 solves this with Dual Representation: the same documents live in a vector database (Postgres + pgvector) for factual lookups and in a Neo4j knowledge graph for relational traversal. An agent then reasons about which store to query.
How do you architect the dual-store system?
Start by classifying your query types into single-entity lookups versus relational/comparison questions. If both exist — and in enterprise settings they almost always do — you need both retrieval tools. Set up Neon (free-tier Postgres with pgvector) and run the SQL schema, updating the vector dimension if you switch embedding models away from text-embedding-3-small's 1536. Deploy Neo4j via Neo4j Desktop or a local AI package and capture the bolt URL and credentials.
Configure your .env with DATABASE_URL, Neo4j credentials, a main LLM provider, an embedding provider (which can differ — useful if you use OpenRouter, which lacks embeddings), and a separate lightweight ingestion LLM like GPT-4.1 Nano to keep extraction costs down across large corpora.
How do you tune the agent's routing for your domain?
The routing logic lives in agent/prompts.py, and this is where enterprise assistants succeed or fail. Do not ship the generic template. Encode explicit rules naming your entity types (vendors, products, teams) and relationship types (integrates_with, acquired_by, reports_to). Tell the agent to use the knowledge graph whenever two entities appear in one question or when the question asks how things relate, and to use vector search for single-entity factual lookups.
Run ingestion with `python -m ingestion.ingest --clean` on a fresh project — never on production, since --clean destroys and recreates tables. Vector ingestion finishes in seconds; graph ingestion via Graffiti takes minutes per document as the ingestion LLM extracts entities and relationships. Monitor episodes in the Neo4j dashboard and never interrupt.
How do you validate before shipping to stakeholders?
Start the FastAPI endpoint (`python -m agent.api`) and the CLI (`python cli.py`). The CLI shows which tools the agent invoked in real time. Run at least one single-entity query, one relational query, and one that should trigger both. Because LLMs are non-deterministic, routing varies run to run — judge on answer quality, not exact tool calls. If the agent consistently routes relational questions to vector search, sharpen the prompt rules rather than touching infrastructure.
For production, front the FastAPI endpoint behind your internal auth, schedule re-ingestion as documents change, and keep the ingestion LLM lightweight to control the front-loaded cost. Querying is fast at runtime, so the cost asymmetry stays at ingestion where you can batch it off-hours.
Next step: Map your organization's ten most common knowledge queries into single-entity versus relational buckets today — that classification determines whether you build vector-only or the full dual-store RAG 2.0 system.
// FREQUENTLY ASKED QUESTIONS
How do I secure the FastAPI agent endpoint for internal use?
Front the FastAPI endpoint behind your existing internal authentication or an API gateway before exposing it to stakeholders. The agent itself has no built-in auth. Restrict network access to the Postgres and Neo4j instances, keep credentials in environment variables never committed to source, and log tool invocations for auditing which knowledge stores answered each query.
How often should I re-ingest enterprise documents?
Re-ingest whenever source documents change materially. Since graph ingestion is minutes per document, schedule it off-hours and consider incremental ingestion rather than full --clean rebuilds on production. Keep --clean strictly for fresh, non-production setups — running it on live data destroys and recreates all tables, wiping your knowledge base.
Can this scale to thousands of enterprise documents?
Yes, but the front-loaded graph ingestion cost grows linearly, so use a lightweight ingestion LLM like GPT-4.1 Nano to keep per-document extraction cheap. Vector ingestion stays fast. At runtime both vector and graph queries are fast regardless of corpus size, so scale your ingestion batching and infrastructure rather than worrying about query latency.