How to Keep Agent Harnesses Current Across Models

Why does a harness that worked last quarter break on the new model?

Because a harness is always right for a specific model generation, not for all time. Every scaffold component exists to compensate for a spiky behaviour — a specific failure mode of the model you built it for. When you upgrade, those failure modes change. Context anxiety that plagued the old model may be gone; the new model may hold coherence across a 2-hour continuous build; it may now willingly discard its own work and restart when the rubric isn't met.

If you leave the old scaffolding in place, you're paying in complexity and cost for problems that no longer exist. Forced session resets designed for severe context rot add unnecessary overhead on a model with strong coherence. Sprint-by-sprint decomposition essential for a weak planner becomes removable with a stronger agentic model.

How do you decide which harness components to strip?

Start by identifying the new model's spiky behaviours empirically — don't assume. Then run a simplified harness and compare output quality and cost against the full version before committing to any removal. A concrete example: a team upgrading to a more agentic model found context anxiety gone, coherence holding across two hours, and self-discarding behaviour present. They moved to a single continuous session with compaction and ran the Evaluator at end-of-full-generation rather than per sprint. The structure stayed Planner-Generator-Evaluator, but with fewer moving parts.

Reassess after every major model release which components are still load-bearing: context-window resets, sprint decomposition, and Evaluator cadence are all candidates that shift generation to generation.

Why is reading traces the core discipline, not running experiments?

Because running more experiments is a false shortcut. The primary debugging loop for any long-running harness is reading agent transcripts by hand, line by line — like reading a stack trace. Only by empathising with why the model made each decision can you know which scaffold components to delete, adjust, or keep as the frontier moves.

Find every point where the Evaluator's judgment diverged from yours, then update the Evaluator's system prompt and rubric to close that gap. To scale this across a platform, pipe transcripts to a secondary agent that greps for patterns and suggests prompt updates — but treat those as candidates for human review, not automatic commits. Human trace-reading stays the ground truth.

How do you keep state reliable across sessions and handoffs?

Use the file system as shared state, not the context window. Persistent artifacts — progress files, feature lists as JSON (never markdown, which models overwrite), and timestamped learnings logs — are more reliable than context-window memory and survive session boundaries, compaction, and human handoff. Compaction does not equal coherence; lossy summaries drift over very long runs, so never rely on it alone for state management.

This file-based discipline also makes your harness portable across model generations: the state layer stays stable even as you swap out scaffolding, so you can experiment with simpler configurations without losing your source of truth.

Next step: Instrument your harness to persist full transcripts, schedule a trace-reading review after your next model upgrade, and run a simplified harness variant side-by-side to measure which scaffold components are still load-bearing on quality and cost.