How to Audit Prompts Before a Model Upgrade
For ML platform teams preparing model upgrades · Based on Bri's Production Prompt Engineering Method
// TL;DR
ML platform teams preparing a model upgrade can use Bri's Production Prompt Engineering Method to prevent regressions before they hit production. Newer models follow instructions more literally and abstain rather than guess, so prohibitions and stale patches that older models glossed over suddenly dominate behavior. The method runs a patch audit, rewrites every prohibition as decision criteria, resolves conflicting instruction pairs, declares exact output shapes, and validates the whole thing against eval tests on the new model. Use it as a mandatory pre-upgrade step on every production prompt.
Why do prompts break when we upgrade the model?
Because newer models follow instructions more literally and will abstain rather than guess. Patterns older models loosely interpreted — vague prohibitions, conflicting patches, unstated output shapes — now get enforced to the letter. A prompt that said 'avoid making assumptions' might have been ignored on the old model but causes wholesale refusals on the new one. Assuming a working prompt will keep working after an upgrade is one of the most expensive pitfalls in production.
That's why a model upgrade is an explicit trigger for Bri's method. Before you swap the model, you audit the prompt so the more literal interpreter behaves predictably.
How do we run a pre-upgrade patch audit?
Go through every patch — every instruction added after the fact to fix a bug — and ask three questions. Is it still relevant, meaning does the underlying bug still exist? Does it conflict with any other instruction? Can you name the specific failure it prevents? If you can't name the failure, that instruction is the first thing to cut. Flag all 'always' and 'never' instructions for this test because they're the ones a literal model will enforce hardest.
Conflicting instruction pairs are the real danger. The model tries to enforce all of them, and no one can predict which wins on a given run — behavior that was masked on the old model becomes chaotic on the new one. Resolve every pair: keep one, remove or merge the other. Remove stale instructions whose root cause was fixed long ago.
How do we prevent over-caution on the new model?
Rewrite every prohibition as decision criteria. Scan for 'never,' 'avoid,' 'don't,' and 'critical,' and convert each into a calm, two-sided rule stating when to act, when to hold back, and why. On a literal model, urgency signaling causes over-caution — the model refuses tasks it's fully equipped to do. Neutral decision criteria let it evaluate the situation instead of reacting to strong-word priority signals.
Also check for one-sided trade-offs. A judgment-call instruction that states only one cost will make a literal model drift hard in one direction. State both costs so the model can weigh them.
How do we validate the upgrade won't regress?
With evals, on the new model, before rollout. Set up an eval suite that makes real API calls against representative inputs and records baseline behavior on the current model. Then point the same suite at the upgrade candidate and compare. Run each test multiple times, since model calls vary, and look for consistent, correctly shaped outputs.
This eval-before-iterate discipline is what turns a risky upgrade into a measured one. Without it, you're guessing whether the new model helped or hurt. With it, you can see exactly which prompts regress and fix them — usually by rewriting a prohibition or resolving a conflict — before any user is affected.
What should the static prompt contain after the audit?
Only stable instructions. Move all dynamic data — record IDs, account details, live pricing — into runtime variables handled by the harness. Offload any deterministic computation to tools or harness code. Declare exact output shapes in reply tags. A lean, structured, patch-free prompt is far more robust across model versions than a heavily patched one, so future upgrades get cheaper each time.
Next step: Inventory every production prompt, tag the ones with the most patches and prohibitions as highest risk, and run the audit-and-eval process on those first before scheduling the model swap.
// FREQUENTLY ASKED QUESTIONS
Do we really need to audit every prompt before upgrading?
Prioritize by risk: prompts with the most patches, the most 'always'/'never' instructions, and the most judgment calls are most likely to regress on a more literal model. Audit those first. At minimum, rewrite prohibitions as decision criteria and re-run evals on the upgrade candidate for every production prompt before rollout.
How do conflicting instructions cause upgrade regressions?
Conflicting instructions direct the agent toward incompatible behaviors. Older models often glossed over one side, but a more literal model tries to enforce all of them, producing unpredictable outputs where no one can predict which instruction wins. Resolving each conflicting pair before the upgrade — keeping one, removing or merging the other — eliminates this source of regression.
What eval metrics matter most for a model upgrade?
Track whether the correct decision was made, whether output matches the declared shape, and whether behavior stays consistent across repeated runs of the same input. Compare the upgrade candidate against your current-model baseline on identical representative inputs. A regression shows up as a changed decision, drifting output length, or new refusals — all fixable before rollout.