Why Senior Engineers Build Flaky AI Agents

Why does my agent feel flaky even though my code is clean?

Because the engineering instincts that make you excellent at deterministic systems actively fight agent reliability. Traditional software rewards rigid control flow, exact-match tests, and exception-based error handling. Agents are non-deterministic — the same input won't always produce the same steps or result. When you apply deterministic reflexes to a probabilistic system, the agent appears broken even when it's working correctly. The Schmid Agent-Ready Engineering Framework names the five specific gaps so you can audit your system against each one.

What are the five habits I need to unlearn?

First, text is your new state. You're probably storing agent state as Booleans, flags, or rigid schemas. Agents reason over semantic meaning, so a churn-flag or a dropdown value discards context the model could use. Replace typed fields with context-carrying text like 'focus on US market, ignore California.'

Second, hand over control. If your orchestration hard-codes step one, step two, step three, you're acting as a traffic controller. Rewrite the logic as a goal plus constraints and become a dispatcher: state the destination and the available transport, then trust the LLM to find the route.

Third, errors are just inputs. Stop treating a failed tool call as a crash that restarts the flow. Feed the error back to the model — 'search failed with timeout; consider alternatives' — and let it reason around the failure while preserving accumulated context.

Fourth, move from unit tests to evals. Your CI is probably asserting exact outputs and failing on functionally correct results. Replace those assertions with evals that measure pass rate across runs against a reliability threshold.

Fifth, agents evolve, APIs don't. Your tools assume the caller has your years of context. The agent sees only schemas and doc strings. Every tool must be self-documenting.

How do I audit my existing agent step by step?

Start by reviewing state management: find every Boolean or rigid structure and ask whether it should be natural-language context. Next, locate hard-coded workflows and rewrite them as goal definitions. Then map every failure point and design error-as-input handling, adding checkpointing for any flow running 5+ minutes so a late failure doesn't restart from zero. After that, convert exact-output tests into evals with a defined threshold — for example, 9 of 10 runs must pass functional criteria — and instrument tracing so you can observe what the agent actually does.

Finally, pull up every function schema the agent can call. For a tool like `delete_item(id)`, ask whether someone with zero codebase context would know what `id` refers to, what deletion implies downstream, and what error states exist. If not, rewrite the doc string and parameter descriptions until the semantic interface is fully self-explaining.

How should I think about architecture longevity?

Apply the build to delete principle. Assume your agent will be rebuilt as models improve, and avoid over-investing in brittle scaffolding tied to a specific model's quirks. Couple your architecture to durable principles — goals, semantic interfaces, eval criteria — rather than workarounds a better model will make obsolete. Flag tightly coupled components as replacement candidates and document them.

Then commit to the observe-adjust loop: define instructions, run, observe via tracing, adjust prompts or tools, run again. This iterative cycle, not a one-shot deploy, is the core development process for agents. Track prompt and tool changes like code changes.

Next step: Take your flakiest agent, run it ten times with tracing on, and measure its actual success rate against a threshold. That single number will tell you whether you have a reliability problem or a testing problem — and the five gaps above will tell you which one to fix first.