Frequently Asked Questions About Anthropic Planner-Generator-Evaluator Long-Agent Framework
21 answers covering everything from basics to advanced usage.
// Basics
What does the Planner actually output?
The Planner outputs three persistent artifacts from a vague one-line prompt: a sprint-level feature list saved as featurelist.json (JSON, not markdown, because models overwrite markdown), a progress file tracking completion state, and a Git repo initialised with an init script so later sessions don't re-derive setup. It keeps the spec deliberately high-level and never makes granular technical decisions.
What is the difference between the Generator and the Evaluator?
The Generator is the builder/IC role that picks one feature at a time and implements it in its own context window. The Evaluator is the adversarial critic/QA role in a separate context window with a harshly-tuned system prompt. The Evaluator uses live tools like Playwright to actively test the artifact — it only sees the output, never the Generator's reasoning trace.
Why should feature lists be JSON instead of markdown?
Models tend to overwrite markdown files but are far less likely to overwrite structured JSON. Since the feature list, progress tracking, and learnings logs are your persistent source of truth across sessions, using JSON protects that state from being clobbered mid-run. The file system — not the context window — is the reliable shared-state mechanism for long-running agents.
What is a contract in this framework and why does it replace the spec?
A contract is a negotiated file-based agreement between Generator and Evaluator that defines what 'done' means for a sprint — around 20-30 granular testable criteria. It replaces the Planner's spec for grading because it converts fuzzy user stories into specific assertions without the Planner over-specifying upfront. The Evaluator grades against this negotiated contract, not the original request.
// How To
How do I set up the three agents with separate context windows?
Instantiate three agents, each with its own context window, system prompt, and single responsibility: Planner (done after decomposition), Generator (builder), Evaluator (critic). Never share the Generator's raw reasoning context with the Evaluator — it muddies adversarial pressure. The Evaluator should receive only the output artifact so its harsh grading isn't softened by the Generator's optimistic narrative.
How do I run the contract negotiation before coding starts?
Have the Generator propose what it will build and how it should be verified. The Evaluator responds via a shared file on disk, pushing back on scope, weak tests, or missed edge cases. They iterate through file read/write until both agree. Target around 20-30 granular criteria — vague criteria produce vague critiques and arbitrary changes.
How do I make the Evaluator actually test the app instead of reading diffs?
Give the Evaluator live verification tools: Playwright MCP for web apps or computer use for native apps. Instruct it to open live pages, click around, drag elements, refresh, and stress-test features against contract criteria. For design quality, have it take screenshots and score against a calibrated rubric. Reading diffs alone lets bugs slip through.
How do I grade subjective qualities like design and originality?
Subjective quality is gradable if you write down a strong enough opinion. Create a rubric with 2-4 dimensions (e.g. Design, Originality, Craft, Functionality), then calibrate the Evaluator with few-shot reference examples labelled 'good design' and 'AI slop.' Weight the rubric toward dimensions where the model is weakest — not functionality if it already handles that well.
What model should I use for each role?
Use your most capable planning model (Opus-class) for the Planner, since decomposition errors cascade over multi-hour runs. Model selection drives harness design overall because capabilities and failure modes differ per generation. The Evaluator benefits from strong reasoning to grade harshly and consistently. Match the Generator to your target domain and iterate based on observed spiky behaviours.
// Troubleshooting
My long-running agent keeps drifting off-track. What's wrong?
You're likely relying on the context window for state and hitting context rot — coherence degrades the deeper the session runs. Move shared state to the file system: progress files, JSON feature lists, timestamped learnings logs. For models with severe rot, start a fresh context window per feature. Don't assume compaction preserves coherence — lossy summaries drift over long runs.
My agent rushes and leaves features half-finished near the end. Why?
That's context anxiety — the model rushes to finish prematurely as it approaches its context limit. Combat it with deliberate context management: use structured hand-offs, fresh sessions per feature, or compaction depending on your model generation. Persistent artifacts let a fresh session pick up cleanly without the pressure of a nearly-full window.
The Generator keeps patching the same broken feature. How do I fix it?
If the Generator can't hill-climb against a criterion after repeated attempts, the harness should discard the current attempt entirely and restart that sprint from scratch — not keep patching. This course-correction over long time horizons is the core advantage over a RALF loop or single-session self-review. Build the discard-and-restart trigger into your feedback loop.
My Evaluator defers bugs instead of blocking on them. What should I do?
Out of the box, LLMs make lenient QA agents that say 'fix in 2 weeks' rather than blocking. This requires significant prompt tuning. Read transcripts where the Evaluator's judgment diverged from yours, add few-shot examples of what must block, and rewrite the system prompt to demand blocking on failed contract criteria. Plan for this calibration work upfront.
// Comparisons
How is this different from just prompting a single coding agent to build carefully?
A single agent can't reliably critique its own work — sycophancy bias makes it call half-baked features done. The framework separates building from judging into distinct context windows with adversarial pressure, and adds file-based state plus discard-and-restart. This produces production-grade output over multi-hour runs that a solo agent loop, however carefully prompted, cannot achieve.
How does the Generator-Evaluator gap relate to GANs?
It borrows the GAN insight directly: tuning a standalone discriminator to be harsh is tractable, while tuning a generator to self-critique is not. It's easier to critique a meal than cook one. The framework exploits this asymmetry by giving the Generator and Evaluator separate context windows, system prompts, and jobs — creating productive adversarial tension analogous to a GAN's two networks.
How does this compare to context-window compaction alone?
Compaction summarises prior context to extend a session beyond the raw limit, but summaries are lossy and drift over very long runs — compaction does not equal coherence. The framework instead uses the file system as the source of truth: JSON feature lists, progress files, learnings logs. Compaction can complement it for capable models, but never replace persistent artifacts.
// Advanced
Should I always use separate sessions per feature?
No — it's model-specific. Fresh sessions per feature help models with severe context rot or anxiety, but for newer models that hold coherence across a 2-hour build, forced resets add unnecessary complexity and cost. Reassess after every major model release which scaffold components are still load-bearing and strip the redundant ones.
How do I know which harness components to remove as models improve?
Identify the current model's spiky behaviours — its specific failure modes — and keep only scaffolding that fills those gaps. Run a simplified harness version and evaluate output quality and cost before committing to removal. Read traces to see whether a component is still load-bearing. The harness co-evolves with the model; it should never be treated as permanent.
Why is reading agent transcripts the primary debugging loop?
Because only by reading transcripts line by line — like reading a stack trace — can you empathise with why the model made each decision. This tells you which scaffold components to delete, adjust, or keep. Running more experiments is a false shortcut; without understanding where the model's judgment diverged from yours, prompt tuning is guesswork.
How many contract criteria should each sprint have?
Target roughly 20-30 granular, actionable criteria per sprint. This range is specific enough to produce meaningful, testable grading without over-specifying. Fewer criteria produce vague critiques and arbitrary Generator changes; far more can bloat the negotiation. Each criterion should describe both the requirement and how to verify it — e.g. 'drag three tasks, refresh, confirm order persists.'
Can I automate the trace-reading and prompt-tuning loop?
Partially. You must read traces by hand initially to understand where judgment diverged, but you can pipe transcripts to a secondary agent to grep for patterns and suggest prompt updates. Treat those suggestions as candidates, not commits. Human trace-reading remains the ground truth for precise tuning — automation accelerates pattern-finding, not judgment.