Nick Nisi Harness Engineering for AI Agents

Last updated: 31 May 2026

Ship reliable AI-agent pipelines by replacing trust with cryptographic evidence, state-machine enforcement, and failure-driven memory — so your agents stop lying and start proving.

// TL;DR

Harness Engineering for AI Agents is a framework for building reliable multi-agent engineering pipelines by replacing trust with proof. Instead of prompting agents to complete steps, you enforce them through a state machine with hard gates, require cryptographic or mechanical evidence (test-output hashes, Playwright videos) for every claimed completion, and run a retrospective agent that writes lessons to memory files after each run. Use it when building or auditing autonomous agent systems that must finish multi-step tasks, when agents hallucinate completion or skip steps, or when writing AI-facing skills and docs for your product.

Framework

// When should you use harness engineering for AI agents?

Use this skill whenever you are building or auditing an AI agent system that must autonomously complete multi-step engineering tasks. Also apply it when your agents are inconsistently completing steps, hallucinating completion, or when you are creating AI-facing documentation/skills for your own product.

// What do you need before building an agent harness?

task_sourcerequired
The triggering work item the agent must act on: a GitHub issue, PR, Linear ticket, Slack thread, or equivalent.
target_codebase_contextrequired
The repo(s), languages, and frameworks the agent will operate in.
definition_of_donerequired
A concrete, verifiable success condition — e.g. 'tests pass', 'UI bug fixed', 'auth installed'. Must be provable without trusting the agent's self-report.
known_gotchas
A list of the specific landmines, implicit contracts, or edge cases in this codebase or product that agents reliably get wrong.
existing_memory_files
Any markdown memory files the harness has previously generated for this project/framework.

// What are the core principles of harness engineering?

Enforce, Don't Instruct

Never rely on a prompt asking the agent to do something. Use a state machine or external pipeline gate to make it structurally impossible for the agent to advance without completing the required action. The agent can ignore instructions; it cannot bypass a hard gate.

Replace Trust with Evidence

Every claimed completion must be cryptographically or mechanically proven — not accepted on the agent's word. If it ran tests, the test output must be SHA-256 hashed into a verifiable artifact. If it fixed a UI bug, it must record a Playwright video before and after. Never waste review time on unproven work.

Guide, Don't Prescribe

Do not feed the agent a comprehensive dump of your docs. Instead, surface only the specific gotchas — the landmines the model reliably hits in your product. The model already knows how to code; it just needs to know where the implicit contracts and failure points are.

Measure, Don't Assume

Run evals before and after every change to skills or prompts. More context does not mean better performance — it can actively degrade it. Trust is a pass rate, a hash, a delta score. If you are not measuring, you are adding noise.

Every Failure Is a Harness Bug

When the agent makes a mistake, do not fix the agent's output — fix the harness so the harness corrects the agent. The system should be self-correcting; your job is to improve the environment, not patch individual outputs.

Retrospective Memory Loop

After every run, a retrospective agent reviews the full execution log (tool calls, loops, retries) and updates per-project markdown memory files with lessons learned. This prevents the agent from hitting the same landmine twice and enables the harness to improve automatically over time.

// How do you build an agent harness step by step?

1
Identify and frame the task from its source artifact
Point the harness at the task source (issue, ticket, Slack thread). The harness — not you — gathers the required context. Do not manually brief the agent; automate context ingestion so setup time is not your bottleneck.
2
Define a provable Definition of Done before execution starts
Specify the evidence artifact required: test output hash, Playwright before/after video, diff of changed files, etc. If the success condition cannot be mechanically verified, redesign it until it can. Ambiguous done-states allow agents to lie.
3
Load relevant gotcha memory files for the target framework/project
Pull only the markdown memory files relevant to the current project and framework. Do not load the entire knowledge base. Irrelevant context degrades performance — send only targeted, landmine-specific guidance.
4
Run the Implementer agent inside the state machine
The implementer attempts the task. It operates inside a controlled loop — it cannot self-report completion or advance. All tool calls and outputs are logged to a structured transcript (e.g. JSONL) for the retrospective agent.
5
Gate on the Verifier before any review
The state machine enforces that the Verifier must confirm the implementation before the Reviewer can run. This is a hard gate — not a prompt instruction. The Verifier checks the cryptographic/mechanical evidence artifact defined in Step 2.
6
Run the Reviewer and loop back to Implementer if issues found
The Reviewer checks code quality. Any issues trigger a mandatory return to the Implementer — the state machine blocks forward progress until the Reviewer is satisfied. This loop continues; the agent cannot escape it by self-asserting done.
7
Run the Closer to attach evidence to the deliverable
The Closer only activates once the Reviewer clears the work. Its job is to package and attach the evidence artifact (video, hash, test output) to the PR or deliverable. No evidence = no PR. Human reviewers should refuse to look at output without this evidence attached.
8
Run the Retrospective agent on the full execution log
The Retrospective agent reads the full JSONL transcript of the run. It identifies doom loops (same tool called 3+ times with no change), skipped steps, and repeated mistakes. It writes lessons to the appropriate per-project or per-framework markdown memory file. This is non-optional — it runs every time, success or failure.
9
Treat every failure as a harness bug and fix the harness
Do not manually patch the agent's output. Identify which gate, memory file, or gotcha was missing that allowed the failure, then update the harness accordingly. The next run should structurally prevent the same mistake — not just hope the agent remembers your feedback.
10
Re-run evals after any change to skills, gotchas, or harness logic
Use an eval suite to measure pass rate before and after the change. If performance drops, revert. More content in skills is not better — 553 lines of targeted gotchas can outperform 10,000 lines of comprehensive docs. Delete anything that reduces pass rate.

// What does harness engineering look like in practice?

An agent is tasked with fixing a UI regression in a React component across multiple repos. The developer wants to verify the fix before reviewing the code.

Apply 'Replace Trust with Evidence': require the agent to use a headless browser tool to record a video of the broken behavior before the fix and the working behavior after. The Closer attaches both videos to the PR. The human reviewer only looks at the PR once the videos prove the fix — otherwise the agent is sent back to the Implementer gate.

A developer is building AI-facing skills/docs for a third-party SDK that has implicit framework contracts agents consistently violate.

Apply 'Guide, Don't Prescribe' and 'Measure, Don't Assume': instead of converting the entire SDK docs into skills, run evals to identify which specific behaviors the model gets wrong. Write a targeted gotchas file (aim for under 600 lines) covering only those failure points. Re-run evals to confirm the gotchas file increases pass rate without introducing new failures. Delete any skill content that reduces pass rate.

An agent running tests keeps advancing in the pipeline without actually executing them.

Apply 'Enforce, Don't Instruct' and 'Replace Trust with Evidence': remove the prompt instruction to run tests. Instead, have the harness require a test artifact file whose content is the SHA-256 hash of the actual test output. The state machine gate checks that this hash exists and is valid before the Verifier can proceed. It is now structurally easier to run the tests than to fabricate the artifact.

// What mistakes should you avoid when building an agent harness?

Instructing the agent to do something instead of enforcing it through a state machine gate — agents will skip, forget, or lie about completing instructed steps.
Accepting the agent's self-report of completion as truth — always require a mechanically verifiable evidence artifact, never a text claim.
Adding more context (longer skills, comprehensive docs) assuming it improves performance — more tokens can actively degrade results; always measure with evals before and after.
Manually fixing the agent's mistakes on a per-run basis instead of fixing the harness — this creates no lasting improvement and keeps you as the bottleneck.
Skipping the retrospective step on successful runs — the retrospective runs every time, not just on failures, to continuously improve memory files.
Building skills by converting comprehensive documentation wholesale — focus exclusively on the gotchas: the specific landmines agents reliably hit in your product.
Assuming the model does not already know how to code your domain — it usually does; your job is to expose the product-specific implicit contracts it cannot know, not to re-teach coding fundamentals.
Never running evals — without measurement, you cannot distinguish between improvements and noise, and you will unknowingly ship degradations.

// What are the key terms in harness engineering?

Harness: The external pipeline, state machine, and tooling environment that wraps and controls agent execution. The harness enforces gates, manages memory, and logs transcripts — it is what you improve when agents fail, not the agent's individual output. Concept attributed to Ryan Leuppolo's Harness Engineering.
Case: The specific harness implementation described by Nick Nisi: a TypeScript state machine orchestrating five agents (Implementer, Verifier, Reviewer, Closer, Retrospective) with enforced gates between each stage.
Gates: Hard enforcement checkpoints between agent stages in the state machine. A gate is not a prompt instruction — it is code that structurally blocks the pipeline from advancing until the required evidence or condition is satisfied.
Gotchas: A targeted, minimal list of the specific landmines, implicit contracts, and common failure points that AI agents reliably get wrong in a particular product or codebase. Gotchas replace comprehensive documentation as the preferred input for agent skills.
Evidence Artifact: A mechanically verifiable proof that an agent completed a required action — e.g. a SHA-256 hash of test output, a Playwright before/after video, a structured diff. Evidence artifacts cannot be fabricated by the agent without actually doing the work.
Retrospective Agent: The final stage of the harness that reads the full execution transcript (JSONL logs) of a completed run, identifies inefficiencies and repeated mistakes, and writes lessons to per-project markdown memory files so future runs avoid the same failures.
Memory Files: Per-project and per-framework markdown files maintained by the Retrospective Agent that store accumulated lessons, gotchas, and hints. Loaded at the start of relevant tasks to give the agent context it has earned through prior failures.
Doom Loop: A failure pattern where an agent calls the same tool three or more times in succession with no meaningful change in state, indicating it is stuck. Detected by the Retrospective Agent in execution logs.
Evals: A structured test suite run against the agent system to measure pass rate on defined tasks, with and without specific skills or context loaded. The mechanism by which 'Measure, Don't Assume' is operationalised — the only reliable way to know whether a change improved or degraded performance.
Trust is a Pass Rate: Nick Nisi's formulation that human trust in an agent system must be grounded in a measurable score (pass rate, delta score, hash verification) — not subjective confidence or the agent's self-assertion.

// FREQUENTLY ASKED QUESTIONS

What is harness engineering for AI agents?

Harness engineering is a framework for building reliable AI-agent pipelines by wrapping agents in an external state machine that enforces steps through hard gates rather than prompts. Every claimed completion must be proven with a mechanical evidence artifact — a test-output hash or Playwright video — and a retrospective agent writes lessons to memory files so the system self-corrects over time.

What is an evidence artifact in an AI agent pipeline?

An evidence artifact is mechanically verifiable proof that an agent actually did the work — for example a SHA-256 hash of real test output, a Playwright before/after video, or a structured diff of changed files. The key property is that the agent cannot fabricate the artifact without performing the task, so you stop trusting self-reports and start verifying provable completion.

How do I stop my AI agent from lying about completing tasks?

Stop instructing the agent and start enforcing through a state-machine gate that structurally blocks progress until a verifiable evidence artifact exists. Instead of prompting it to run tests, require a file containing the SHA-256 hash of real test output; the gate checks the hash before advancing. It becomes easier for the agent to run the tests than to fake the artifact.

How do I build a harness for multi-step agent tasks?

Point the harness at a task source, define a provable Definition of Done, load only relevant gotcha memory files, then run the Implementer inside a state machine that gates on a Verifier before a Reviewer, loops back on issues, and uses a Closer to attach evidence. A Retrospective agent then reads the full log and updates memory files every run.

How does harness engineering compare to just prompting an agent better?

Prompting asks the agent to behave and hopes it complies; harness engineering makes misbehavior structurally impossible. Agents can ignore, forget, or lie about instructions, but they cannot bypass a code-level gate or fabricate a cryptographic hash. Better prompts degrade under context pressure, while a harness enforces completion and measures every change with evals, giving you durable reliability instead of fragile hope.

When should I use harness engineering instead of a simple agent loop?

Use it whenever an agent must autonomously complete multi-step engineering tasks, when agents inconsistently finish steps or hallucinate completion, or when you are authoring AI-facing documentation and skills for your product. A simple loop is fine for trivial, easily-checked tasks, but anything that touches real code, must pass review, or requires proof of correctness needs enforced gates and evidence.

What results can I expect from harness engineering?

Expect agents that stop hallucinating completion, reviewers who never waste time on unproven work, and a pipeline that self-corrects instead of requiring per-run patching. Because every change is measured with evals, you can distinguish real improvements from noise — and targeted gotcha files (often under 600 lines) can outperform 10,000 lines of comprehensive docs on pass rate.

What is the difference between gotchas and comprehensive documentation for agents?

Gotchas are a minimal, targeted list of the specific landmines and implicit contracts agents reliably get wrong in your product; comprehensive docs are everything. The model already knows how to code, so dumping full docs adds noise that can degrade performance. Gotchas surface only the failure points the model cannot know, keeping context lean and pass rates high.

Why should every agent failure be treated as a harness bug?

Because fixing an agent's individual output creates no lasting improvement and keeps you as the bottleneck. When an agent fails, identify which gate, memory file, or gotcha was missing that allowed the failure, then fix the harness so the next run structurally prevents it. The goal is a self-correcting environment, not patched one-off outputs.

What are evals and why are they essential for agent systems?

Evals are a structured test suite that measures your agent system's pass rate on defined tasks, with and without specific skills or context. They operationalize 'measure, don't assume' — the only reliable way to know if a change improved or degraded performance. Without evals you cannot tell improvements from noise and will unknowingly ship regressions when adding context.

// GET THIS SKILL — FREE