How to Build a Runbook Search Bot with Agentic RAG
For Engineering teams at SaaS companies · Based on Ebbelaar Agentic RAG From Scratch
// TL;DR
Engineering teams can use Ebbelaar Agentic RAG From Scratch to turn a folder of internal runbooks and architecture decision records into a natural-language Slack bot. The LLM sits inside a loop with List, Grab, and Read tools, self-correcting until it finds the right runbook section, then returns an answer plus citations (file name and line number) that link engineers straight to the source. Use it when your team needs accurate, verifiable answers over private engineering knowledge and can tolerate multiple LLM calls per query. Skip it for latency-critical paths where semantic RAG is faster and cheaper.
Why do engineering teams need Agentic RAG for runbooks?
Most SaaS teams accumulate dozens of runbooks, architecture decision records, and incident postmortems as scattered markdown files. When an on-call engineer needs the deployment window or the database migration procedure at 2am, semantic RAG often surfaces the wrong chunk because it retrieves in a single pass and can't self-correct. Agentic RAG places the LLM inside a feedback loop with three tools — List Files, Grab, and Read Files — so it can re-search when the first attempt misses, iterating until it finds the exact runbook section. More LLM calls means higher accuracy, which matters far more than speed during an incident.
How do you wire runbooks into an Agentic RAG Slack bot?
Point your notes directory at the runbooks folder of markdown files. Build the three core tools against it: List Files using pathlib glob returning relative paths, Grab using ripgrep to search across all files for terms like 'deployment window' or 'rollback', and Read Files with a `read_max_lines` cap and a path containment check. Wire all three to GPT-4o inside an agentic loop with a 10-iteration request limit. Pass the engineer's Slack message as the user query, and let the model autonomously decide which tools to call.
How do you make the bot's answers verifiable?
Define a structured output schema with an `answer` field and a `citations` list, where each citation holds the file name, exact quote, and line number. The Slack bot then posts the plain-English answer alongside links that jump directly to the relevant runbook section. This audit trail is critical for engineering — nobody wants to act on an unsourced answer during an outage. Citations turn the bot from a guessing machine into a trusted first responder that always shows its work.
How do you tune retrieval for engineering vocabulary?
Enable streaming steps during QA to intercept every tool call. For each test query, log which tool the model called, what search pattern it chose, and what results came back. If the model searches 'deploy' but your runbooks say 'deployment window', refine the Grab docstring with domain-specific hints so the model generates better patterns. Docstrings are prompt engineering — the richer they are, the more reliably the bot surfaces the right file and line. Iterate query by query until your common on-call questions consistently return the correct source.
What should you harden before shipping to production?
Add path containment checks (`is_relative_to`) on every read so the agent can never escape the runbooks folder into host files. Set the agent request limit to stop runaway loops. Enforce the `read_max_lines` cap so a giant postmortem doesn't blow up the context window. Return human-readable error strings from every tool instead of raising exceptions, so a missing file lets the agent retry rather than crash the bot. Replace the Python regex Grab with ripgrep for speed and .gitignore-awareness, and add a startup check confirming ripgrep is installed in your container.
What results can an engineering team expect?
Engineers get accurate, cited answers from private runbooks in natural language, without hunting through a wiki mid-incident. Because the agent self-corrects, it handles vague or misspelled questions far better than semantic RAG. The trade-off is higher per-query latency and cost from multiple LLM calls — acceptable for a runbook bot but not for latency-critical automation.
Next step: Export your runbooks to a /notes markdown folder, build the three tools against it, and wire them to your Slack bot with a structured citation schema before your next on-call rotation.
// FREQUENTLY ASKED QUESTIONS
How many runbooks can Agentic RAG handle?
Agentic RAG scales well beyond what fits in a single prompt because it only reads the specific files and lines the agent needs. Fifty runbooks is trivial, and with ripgrep handling the search subprocess, hundreds of files stay fast. The limiting factor is per-query cost and latency from the loop's multiple LLM calls, not the size of the knowledge base itself.
Can the Slack bot link directly to a runbook section?
Yes. The structured output schema returns citations with file name and line number, so your bot can construct a link that jumps to the exact section in your wiki or repo viewer. This gives engineers a verifiable source they can open in one click rather than trusting an unsourced answer during an incident.
Is Agentic RAG safe to point at our private engineering docs?
Yes, provided you add path containment checks using `is_relative_to(notes_dir)` on every read. This confines the agent to your runbooks folder and prevents it from being prompted or hallucinating its way into reading arbitrary host files. Combined with request limits and error-returning tools, the agent stays safely sandboxed to the knowledge you designate.