How to Build Coding Agents That Run for Hours
For AI engineers building autonomous coding agents · Based on Anthropic Planner-Generator-Evaluator Long-Agent Framework
// TL;DR
The Planner-Generator-Evaluator framework lets AI engineers build coding agents that run autonomously for hours and ship complete multi-feature apps. It splits work across three agents with separate context windows — a Planner that decomposes vague prompts into sprints, a Generator that builds one feature at a time, and an adversarial Evaluator that harshly tests output with Playwright. Combined with file-based state (JSON feature lists, progress files, learnings logs), it beats RALF loops and single-agent self-review by course-correcting over long horizons. Use it whenever your agent needs to run longer than ~30 minutes without losing coherence, quality, or direction.
Why do solo coding agents lose the plot after 30 minutes?
Because models can't reliably judge their own output. The same sycophancy and generosity bias that plagues LLM-as-judge systems applies to coding agents — a builder reviewing its own work sees a half-implemented button and calls it done. On top of that, coherence degrades as the session deepens (context rot), and near the context limit the model rushes to finish prematurely (context anxiety). A single agent loop compounds all three failure modes over a multi-hour run.
The Planner-Generator-Evaluator framework solves this by separating building from judging. You instantiate three agents, each with its own context window, system prompt, and single responsibility. The Generator builds; the Evaluator critiques harshly in isolation; the Planner sets high-level direction and then steps back.
How do you architect the three-role harness?
Start with the Planner — use your most capable planning model (Opus-class). Feed it a deliberately vague one-line prompt like 'build a full-stack project management tool.' It outputs three persistent artifacts: a sprint-level `featurelist.json` (JSON, not markdown, because models overwrite markdown), a progress file, and a Git repo with an init script. Keep the spec high-level — over-specifying granular technical decisions here causes errors to cascade and magnify across every sprint.
Before any code is written, the Generator and Evaluator negotiate a contract via shared files on disk: around 20-30 granular, testable criteria defining what 'done' means for the sprint. This contract — not the Planner's spec — becomes the ground truth for grading. The Generator then picks exactly one incomplete feature, orients itself using the progress file and init script, builds it with programmatic tool calling to conserve context, and writes timestamped learnings to a JSON log.
How does the Evaluator actually catch bad output?
By actively using the artifact, not reading diffs. For web apps, the Evaluator launches Playwright MCP to open live pages, click around, drag elements, and refresh to stress-test persistence. For native apps, it uses computer use. It grades against the negotiated contract across your rubric dimensions — and critically, it only sees the output, never the Generator's reasoning trace, which would otherwise soften its judgment with an optimistic narrative.
When the Generator can't hill-climb against a criterion after repeated attempts, the harness discards the current attempt entirely and restarts that sprint from scratch. This discard-and-restart capability over long horizons is the core advantage over a RALF loop (which has a fixed plan and no adversarial pressure) and over single-session self-review.
How do you tune and evolve the harness?
After each run, read the full transcripts by hand, line by line — this is your primary debugging loop, not running more experiments. Find every point where the Evaluator's judgment diverged from yours, empathise with why the model made each decision, and update the Evaluator's system prompt and rubric to close the gap. You can pipe transcripts to a secondary agent to grep for patterns and suggest updates.
Remember the harness co-evolves with the model. Identify your current model's spiky behaviours and fill those gaps with scaffolding — then strip components as the model internalises them. Context resets between sessions may be critical for one generation and redundant the next. Run a simplified version and evaluate before committing to any removal.
Next step: Draft your quality rubric with 2-4 opinionated dimensions, write your Planner prompt to emit `featurelist.json`, and stand up your Generator and Evaluator in separate context windows. Then run one sprint end-to-end and read every line of the transcript.
// FREQUENTLY ASKED QUESTIONS
What model should I use for the Planner versus the Generator?
Use your most capable planning model (Opus-class) for the Planner, because decomposition errors cascade and magnify over a multi-hour run. The Generator should match your target domain, and both Generator and Evaluator benefit from strong reasoning. Model selection drives your entire harness design, since capabilities and spiky failure modes differ per generation.
Should the Evaluator see the Generator's code and reasoning?
No — the Evaluator should see only the output artifact, never the reasoning trace. Sharing the Generator's narrative muddies adversarial pressure and lets the model kid itself that something works based on how it was described rather than how it behaves. Keep the Evaluator's grading grounded in live testing of the actual artifact.
How do I keep state coherent across a multi-hour run?
Use the file system as the source of truth, not the context window. Save the feature list as JSON, track completion in a progress file, and write timestamped learnings logs. Don't rely on compaction alone — lossy summaries drift over long runs. For models with severe context rot, run a fresh session per feature and rehydrate from the persistent artifacts.