How to Verify AI Agent Work Before It Ships
For QA and release engineers adopting AI agents · Based on Nick Nisi Harness Engineering for AI Agents
// TL;DR
If you are a QA or release engineer letting AI agents touch your codebase, harness engineering gives you a verification-first workflow. Never accept an agent's self-report — require a mechanical evidence artifact like a SHA-256 hash of real test output or a Playwright before/after video, and enforce it with a hard Verifier gate the agent cannot bypass. A Closer attaches the proof to every deliverable, and reviewers refuse anything without it. Use it to guarantee that agent-produced changes are verified before they reach release.
Why can't you trust an agent's 'done'?
Because an agent's claim of completion is just text, and text can be fabricated. Agents regularly report that tests passed when they never ran, or that a bug is fixed when it still reproduces. As a QA or release engineer, accepting self-reports means auditing work that may be fiction. Harness engineering's answer is blunt: replace trust with evidence. Every claimed completion must be mechanically proven, or it does not count — and the agent cannot advance until it is.
What makes an evidence artifact trustworthy?
An evidence artifact is proof the agent cannot fabricate without actually doing the work. If it ran tests, require a file whose content is the SHA-256 hash of the real test output — the hash only exists if genuine output exists. If it fixed a UI regression, require a headless-browser recording of the broken behavior before and the working behavior after. A structured diff of changed files works too. The property that matters is mechanical verifiability: you check the artifact, not the agent's story.
How do you enforce verification so it can't be skipped?
Use a hard gate in the state machine. The Verifier stage must confirm the evidence artifact before the Reviewer can run — and this is code, not a prompt instruction. If the hash is missing or invalid, the pipeline simply does not advance. For a UI regression, the human reviewer only opens the PR once the before/after videos prove the fix; otherwise the agent is sent back to the Implementer gate. The agent cannot self-assert past verification.
Who attaches the proof to the final deliverable?
The Closer stage, which activates only after the Reviewer clears the work. Its sole job is to package and attach the evidence artifact — the video, the hash, the test output — to the PR or deliverable. The rule for your team is simple: no evidence, no PR. Human reviewers should refuse to even look at output without attached evidence. This keeps your review queue free of unproven work and makes every merged change independently auditable.
How do you stop the same defects from recurring?
Treat every failure as a harness bug and lean on the retrospective memory loop. When something slips through, do not just file the bug — identify which gate or gotcha was missing and strengthen the harness so the failure is structurally impossible next time. After every run, the Retrospective agent reads the JSONL transcript, flags doom loops and skipped verification steps, and records lessons to memory files. Then run evals before and after any change to prove reliability moved in the right direction.
Next step
Choose your highest-risk agent-produced change type and define its Definition of Done as a mechanically provable artifact. Add a Verifier gate that checks it, and a policy that no PR reaches review without the Closer-attached evidence. Then build a small eval suite so you can report verified pass rate as your release-readiness metric.
// FREQUENTLY ASKED QUESTIONS
What evidence artifact should I require for a bug fix?
Require proof that cannot be faked without doing the work: a Playwright or headless-browser video showing the broken behavior before and the working behavior after, plus a SHA-256 hash of the passing test output. The Closer attaches both to the PR, and reviewers only look once the evidence proves the fix — otherwise the agent returns to the Implementer gate.
How do I enforce that tests actually ran before release?
Stop instructing the agent to run tests and instead require an artifact file containing the SHA-256 hash of the real test output. Add a state-machine gate that validates the hash before the Verifier proceeds. Because the hash can only come from genuine output, it becomes structurally easier for the agent to run the tests than to fabricate the artifact.
What should my release-readiness metric be with AI agents?
Use verified pass rate from an eval suite plus evidence-artifact coverage — the fraction of deliverables shipping with a valid hash or video. 'Trust is a pass rate' means readiness is a measurable number, not a judgment call. Run evals before and after any harness change so you can prove reliability improved and catch regressions before release.