How to Build Reliable AI Agent Pipelines at a Startup
For AI engineering leads at startups · Based on Nick Nisi Harness Engineering for AI Agents
// TL;DR
If you lead AI engineering at a startup and your agents inconsistently finish multi-step tasks or hallucinate completion, harness engineering gives you a reliability framework. You wrap agents in a state machine with hard gates, require mechanical evidence (test-output hashes, Playwright videos) for every completion, and run a retrospective agent that updates memory files after each run. This replaces trust with proof and per-run firefighting with a self-correcting system — so your team ships autonomous agents that prove their work rather than claim it.
Why do startup agent pipelines break in production?
Most startup agent systems are built on prompts and hope. You ask the agent to run tests, review the diff, and only then open a PR — and it works in the demo. Then it ships, context grows, and the agent starts skipping steps, hallucinating completion, and quietly lying about work it never did. The root cause is architectural: an instruction is a suggestion the agent can ignore. Harness engineering fixes this by making the right behavior structurally mandatory instead of merely requested.
How does a state machine make agents reliable?
Instead of a single agent loop, you run five roles inside a TypeScript state machine: Implementer, Verifier, Reviewer, Closer, and Retrospective. Between each stage sits a gate — not a prompt, but code that blocks forward progress until a required condition is met. The Verifier must confirm mechanical evidence before the Reviewer runs; the Reviewer loops work back to the Implementer until quality passes; the Closer only activates once review clears. The agent cannot self-assert its way past any gate, which is exactly what you need when you cannot babysit every run.
What evidence should your agents be forced to produce?
Apply replace trust with evidence: every claimed completion needs a mechanically verifiable evidence artifact. If the agent ran tests, require a file whose content is the SHA-256 hash of the real test output, and gate on that hash. If it fixed a UI bug, require Playwright before/after videos the Closer attaches to the PR. Tell your human reviewers to refuse any deliverable without attached evidence — no video or hash means no review. This single policy eliminates the wasted cycles your team currently spends auditing unproven work.
How do you stop fixing the same failures over and over?
Treat every failure as a harness bug. When an agent fails, do not patch its output — that keeps you as the bottleneck and creates no lasting improvement. Instead, find which gate, gotcha, or memory file was missing and fix the harness so the next run structurally prevents the failure. Pair this with the retrospective memory loop: after every run, a Retrospective agent reads the full JSONL transcript, spots doom loops and skipped steps, and writes lessons to per-project markdown memory files. Load only the relevant files at task start — irrelevant context degrades performance.
How do you prove a change actually helped?
Run evals before and after every change to skills, gotchas, or harness logic. More context is not better; 553 lines of targeted gotchas can beat 10,000 lines of docs. Measure pass rate, keep what raises it, and revert anything that drops it. This is how you report reliability to your team and investors as a number — a pass rate, a delta score — rather than a vibe. Trust becomes a metric you can defend.
Next step
Pick your most failure-prone agent task, rewrite its Definition of Done so success is mechanically provable, add one hard gate plus one evidence artifact, and stand up a minimal eval suite to measure the before/after pass rate. Expand to the full five-agent state machine once that single gate proves out.
// FREQUENTLY ASKED QUESTIONS
How much engineering effort does a harness take to build?
You can start small: rewrite one task's Definition of Done to be provable, add a single hard gate plus one evidence artifact, and build a minimal eval suite. That alone eliminates the most common failure. The full five-agent state machine is an incremental expansion from there — you do not need to build all five roles before seeing reliability gains.
Will harness engineering slow down my team's shipping velocity?
Initially there is setup cost, but it increases net velocity by eliminating per-run firefighting and wasted reviews of unproven work. Because the harness self-corrects through the retrospective loop and prevents repeat failures structurally, your team stops re-solving the same problems. Trust becomes a measured pass rate, so you ship with confidence instead of manual verification.
What should I measure to report agent reliability to leadership?
Report pass rate from your eval suite and the delta between before and after each change. 'Trust is a pass rate' means reliability is a defensible number, not subjective confidence. Supplement with evidence-artifact coverage (what fraction of deliverables ship with a verified hash or video) to show that claimed work is provably real.