Frequently Asked Questions About Tejas Agent Harness Engineering Framework
22 answers covering everything from basics to advanced usage.
// Basics
What exactly is the difference between the agent loop and the harness?
The agent loop is just one component inside the harness — the inner while-true loop that sends a prompt, gets a response, checks for a STOP signal, and pushes events to a trace. The harness is everything around the model: the loop wrapper, the tool registry, guardrails, context management, deterministic handlers, and the verify step. Treating the loop as the whole harness is a common mistake.
What is a trace and why does it matter?
A trace is the accumulated history of all tool calls, messages, and events produced during an agent loop run. It matters because the verify step and deterministic handlers read the trace to make decisions based on evidence rather than querying the non-deterministic model. Without a trace you can't deterministically prove success, detect failed logins, or diagnose where a run went wrong.
Who is harness engineering actually for?
It's for everyone who isn't a 'token billionaire' — Tejas's term for engineers at places like Anthropic or Google with essentially unlimited model access. If you pay rent for compute and must do more with less, the harness lets you use cheap, small, or swappable models reliably. It's the discipline of extracting production reliability without unlimited frontier-model budget.
What does 'fail correctly before you succeed' mean?
It means the first job of a harness is to force the agent to report failure truthfully, because step one to solving a problem is admitting you have one. A verify step that inspects the trace removes the model's ability to lie about success. Only once the lie is gone can you see the real failure and address its root cause with a handler or guardrail.
// How To
How do I build a minimal agent loop to start?
Create a while-true loop that gets a model response; if it returns STOP, return the result, otherwise push the event to a trace. Attach a tool registry where each tool has a name, description, parameters, and an execute function. Use an existing SDK like OpenAI's tool-calling SDK rather than inventing the interface. Log everything to the trace — you'll need that history for the verify step.
How do I add context management without a fancy compressor?
Start naive: when the max_messages guardrail triggers, always keep the system prompt, the user prompt, and the last two messages, then discard everything in the middle. Ship that first, confirm the harness works, then improve the compressor incrementally. Waiting for a perfect compressor before shipping is a pitfall — a crude one prevents context overflow immediately.
How do I write a verify step for a browser task?
Inspect the trace and check concrete evidence. For an upvote task: was there a click on the upvote element, is the current page state consistent with success, and were there no failed-login events? Return early with failure for each known failure pattern — failed login, unexpected redirect. The function returns a definitive pass or fail in code, never asking the model if it succeeded.
How do I structure the harness so it's reusable?
Move all agent logic into run_harness_attempt, then wrap it in run_harness — a retry loop bounded by max_attempts, say three tries. Your entry point shrinks to about 20 lines: define the prompt, call run_harness. This encapsulation makes the harness reusable and composable across tasks and models, and keeps the retry logic separate from the single-attempt loop.
How do I handle an auth wall automatically?
Write a login_handler that fires every loop iteration before the trace updates. It checks the browser's current URL; if it detects a login page, it injects credentials from environment variables and submits the form programmatically — no model involvement. Then it injects a message into the agent's queue: 'Harness: I logged in. You are clear to proceed.' The secret never touches the model context.
// Troubleshooting
My agent keeps claiming the form was submitted but it wasn't. What's wrong?
The agent likely hit a login redirect and lied about success — a classic non-deterministic failure. Fix it in two parts: add a login_handler that detects the redirect and submits credentials deterministically, and add a verify step that checks the trace for a confirmed form-submission tool call AND the absence of login-redirect events. Do not change the system prompt; extend the harness.
My agent loops forever and burns tokens. How do I stop it?
Add a max_iterations guardrail that kills the run after N tool calls — around six for a short browser task. This prevents infinite loops and runaway spend without any prompt change. If runs still blow up on context, also add max_messages with a naive compressor. Guardrails are the harness's job precisely because the model can't be trusted to stop itself.
My harness works on GPT-4 but breaks on a cheaper model. What did I do wrong?
You probably built the harness for one specific model instead of making it model-agnostic. The whole point is that a cheap or swappable black-box model can be made reliable by the surrounding harness. Keep the verify step, handlers, and guardrails model-independent, rely on a standard tool-calling SDK, and don't encode assumptions about one model's behaviour into your deterministic code.
The verify step passes but the task clearly failed. How do I fix it?
Your verify logic is checking the wrong evidence. Read the actual trace from a failed run, find the real signal of failure — a redirect, a missing tool call, an error event — and add that as an explicit failure case that returns early. Verify steps improve by accumulating known failure patterns from real traces, not by guessing. Every missed failure is a new verify case.
// Comparisons
How is a harness different from just adding retries?
Retries alone re-run the same non-deterministic failure and hope for luck. A harness adds a retry loop bounded by max_attempts but combines it with a deterministic verify step that decides whether to retry, plus handlers that fix root causes like auth walls. Retries without verification can retry a 'success' that's actually a lie — the harness proves success before it stops.
How does harness engineering compare to fine-tuning a model?
Fine-tuning changes the probabilistic model and still gives you a non-deterministic system that can hallucinate on critical steps. Harness engineering leaves the model alone and makes critical steps deterministic in code — auth, secret injection, verification. It's cheaper, faster to iterate, model-agnostic, and guarantees correctness where it matters. Fine-tuning improves averages; a harness enforces guarantees.
Why not just use a bigger frontier model and skip the harness?
Because reliability, not raw model quality, is the game. A frontier model running unharnessed still lies about success, leaks secrets, and loops. A cheap model inside a strong harness outperforms it on dependability and costs far less. Frontier access is also finite for everyone outside the token billionaires — the harness lets you do more with less.
Is a harness better than a multi-agent orchestration framework?
They solve different problems. Multi-agent frameworks coordinate many models; a harness makes any single model reliable. You can and should harness the agents inside an orchestration framework. The harness's deterministic verify steps and handlers give each agent the grounding orchestration frameworks assume but rarely enforce, so the two are complementary rather than competing.
// Advanced
What's the enterprise security benefit of the harness owning secrets?
The harness injects secrets and auth tokens deterministically into retrieval or API tool calls, never placing them in the model's context window, so the model can't leak them. A verify step can confirm that retrieved chunks match the authorised user's data scope before returning an answer. This gives enterprise-grade data-access control while the model stays a replaceable black box renter.
What is a dynamic on-the-fly harness?
It's Tejas's predicted next evolution: an agent that, before executing a task, autonomously generates its own harness — identifying where it might hallucinate or fail, creating appropriate guardrails and verify steps — then executes the harnessed plan and returns a guaranteed result. Instead of engineers hand-building harnesses, the agent builds its own deterministic scaffolding per task.
How do I decide what gets a deterministic handler versus the model?
Any step that must be correct goes to a deterministic handler: authentication, secret injection, form submission, result verification. Anything that benefits from flexible reasoning or exploration stays with the model. The rule is deterministic beats probabilistic for critical steps — if a wrong outcome is unacceptable, write code; if variability is tolerable, let the model handle it.
Can I use harness engineering for a RAG pipeline, not just browser agents?
Yes. For an enterprise RAG agent, the harness owns all secrets and auth tokens and injects them into retrieval tool calls deterministically. Guardrails enforce max context size with a compressor, and a verify step confirms retrieved chunks match the authorised user's data scope before the answer returns. The same anchor-the-black-box principle applies to any agentic workflow touching external systems.
How do I iterate on a harness after the first version fails?
Run the harnessed agent, read the trace to find exactly where it went wrong, and add a new guardrail, handler, or verify case for that failure. Repeat. Keep the prompt constant throughout, and judge success only by the verify step's deterministic output, not the model's self-report. Each real failure becomes a permanent, reusable piece of the harness.