Frequently Asked Questions About Nick Nisi Harness Engineering for AI Agents
21 answers covering everything from basics to advanced usage.
// Basics
What is a state machine in the context of AI agents?
A state machine is external code that controls agent execution by defining discrete stages and the hard gates between them. In harness engineering it orchestrates five agents — Implementer, Verifier, Reviewer, Closer, Retrospective — and structurally blocks the pipeline from advancing until each stage's required evidence or condition is satisfied. The agent operates inside it and cannot self-report its way forward.
What are the five agents in the Case harness implementation?
The Case implementation, from Nick Nisi, is a TypeScript state machine orchestrating five roles: the Implementer attempts the task, the Verifier confirms evidence, the Reviewer checks code quality, the Closer attaches evidence to the deliverable, and the Retrospective agent reads the full log and updates memory files. Hard gates sit between each stage so none can be skipped.
What is a doom loop and how is it detected?
A doom loop is a failure pattern where an agent calls the same tool three or more times in a row with no meaningful change in state, meaning it is stuck. The Retrospective agent detects doom loops by reading the structured JSONL execution transcript after a run, then writes a lesson to the relevant memory file so future runs avoid the same trap.
What does 'trust is a pass rate' mean?
It means human trust in an agent system must be grounded in a measurable score — a pass rate, a delta score, or a hash verification — not subjective confidence or the agent's self-assertion. You do not believe the agent finished because it says so; you believe it because an eval passed or a cryptographic artifact verified. Trust becomes a number, not a feeling.
// How To
How do I define a provable Definition of Done?
Specify the exact evidence artifact that proves success — a test-output hash, a Playwright before/after video, a diff of changed files — and verify it is mechanically checkable before execution starts. If the success condition cannot be mechanically verified, redesign it until it can. Ambiguous done-states let agents claim completion without proof, so eliminate ambiguity at the design stage.
How do I write an effective gotchas file?
Run evals to identify exactly which behaviors the model gets wrong in your product, then write a targeted file covering only those failure points — aim for under 600 lines. Do not convert full docs. Re-run evals to confirm the file raises pass rate without introducing new failures, and delete any content that reduces pass rate.
How do I set up the retrospective loop?
After every run — success or failure — have a Retrospective agent read the full JSONL transcript of tool calls, loops, and retries. It identifies doom loops, skipped steps, and repeated mistakes, then writes lessons to the appropriate per-project or per-framework markdown memory file. Make it non-optional so the harness improves automatically over time.
How do I verify an agent actually ran tests instead of faking it?
Remove the prompt instruction to run tests and instead require an artifact file whose content is the SHA-256 hash of real test output. Add a state-machine gate that checks the hash exists and is valid before the Verifier proceeds. Because the hash can only be produced from genuine output, it becomes structurally easier to run the tests than to fabricate the artifact.
// Troubleshooting
Why is my agent still skipping steps even with detailed instructions?
Because instructions are suggestions an agent can ignore, forget, or lie about completing. Detailed prompts do not fix this — enforcement does. Convert the skipped step into a hard gate in the state machine that blocks forward progress until a verifiable condition is met. If the agent cannot advance without producing proof, it cannot skip the step.
My agent's performance got worse after I added more documentation — why?
More context does not mean better performance; it can actively degrade results by adding noise the model must sift through. The model already knows how to code — it needs only your product-specific gotchas. Run evals before and after any content change, revert if pass rate drops, and remember that 553 lines of targeted gotchas can beat 10,000 lines of comprehensive docs.
My reviewers keep wasting time on unproven agent work — how do I fix it?
Instruct human reviewers to refuse any deliverable without an attached evidence artifact: no video, hash, or test output means no review. Enforce this with the Closer stage, which only activates after the Reviewer clears work and whose job is to attach the proof. No evidence equals no PR, so unproven work never reaches a human.
I keep manually fixing my agent's mistakes — what should I do instead?
Stop patching outputs and treat every failure as a harness bug. Identify which gate, memory file, or gotcha was missing that allowed the failure, then fix the harness so the next run structurally prevents it. Manual patching creates no lasting improvement and keeps you as the bottleneck. Re-run evals afterward to confirm the fix raised pass rate.
// Comparisons
How does harness engineering compare to frameworks like LangChain or CrewAI?
Orchestration frameworks like LangChain or CrewAI help you wire agents together but still largely rely on prompts and trust between stages. Harness engineering adds the missing enforcement layer: hard state-machine gates, mandatory cryptographic evidence artifacts, eval-driven measurement, and a retrospective memory loop. You can implement harness principles on top of these frameworks, but the principles are what make the pipeline reliable.
How is harness engineering different from writing better prompts?
Prompt engineering optimizes what you ask; harness engineering changes what is structurally possible. A better prompt still depends on the agent choosing to comply and can degrade as context grows. A harness makes non-compliance impossible through gates and makes fabrication impossible through evidence, and it measures every change with evals rather than assuming the prompt helped.
Is harness engineering overkill for a single simple agent task?
For a trivial task whose output you can glance at and verify instantly, a full five-agent harness is overkill. The principles still apply at smaller scale — define a provable done-state and prefer evidence over trust. The full machinery pays off when tasks are multi-step, touch real code, require review, or when agents inconsistently complete or hallucinate completion.
How does 'guide, don't prescribe' differ from standard RAG over docs?
Standard RAG retrieves and feeds broad documentation into context, assuming more information helps. 'Guide, don't prescribe' deliberately surfaces only the gotchas — the implicit contracts and failure points the model cannot know — because the model already knows how to code. It treats comprehensive dumps as noise and uses evals to confirm that targeted guidance outperforms broad retrieval.
// Advanced
How do I structure memory files across multiple projects and frameworks?
Maintain per-project and per-framework markdown memory files that the Retrospective agent updates after each run. At the start of a task, load only the files relevant to the current project and framework — never the entire knowledge base. Irrelevant context degrades performance, so targeted loading keeps the agent focused on the specific landmines it needs for this task.
Can I use harness engineering to build AI-facing skills for my own SDK?
Yes — it is an ideal use case. Instead of converting your SDK docs wholesale, run evals to find which behaviors the model gets wrong, then write a focused gotchas file (under 600 lines) covering those implicit framework contracts. Re-run evals to confirm the file raises pass rate without new failures, and delete anything that lowers the score.
How do I prevent context degradation while still giving agents enough information?
Treat context as a liability you must justify, not an asset you accumulate. Load only the gotcha memory files relevant to the current project, keep gotchas minimal and failure-specific, and gate information inclusion behind evals. If adding content does not raise pass rate, remove it. The goal is the smallest context that maximizes the measured score.
What should the JSONL execution transcript capture for retrospectives?
The transcript should log every tool call and its output, loops, retries, and state transitions in structured JSONL so the Retrospective agent can analyze the full run. This lets it detect doom loops (same tool 3+ times with no change), skipped steps, and repeated mistakes, then write precise lessons to memory files. Rich logs make the self-correction loop effective.
How do I know when to add a new gate versus a new gotcha?
Add a gate when the failure is a skipped or unproven action the agent can bypass — gates enforce that something must happen and be verified. Add a gotcha when the failure is a knowledge gap about an implicit product contract the model cannot know. Gates make actions mandatory; gotchas correct understanding. Always confirm the choice with eval deltas.