Run It Again Before You Rebuild the Harness

It tells you how much the result can move without your proposed improvement. A harness is the software around a model that supplies instructions, tools and working context. A benchmark is a defined set of tasks with a scoring rule. Before comparing harnesses, establish what happens when the same harness attempts the same benchmark again.

The study behind The Decoder's report on coding-agent harnesses makes that missing comparison explicit. In What Does a Harness Buy? Tokens, Mostly, the researchers found that changing harnesses on their hard task set changed task outcomes about as often as rerunning the same setup. Costs differed even when the measured success rates didn't clearly separate.

That finding has a limited scope: the models, harnesses and tasks tested. It doesn't establish that every harness is interchangeable. It does make a score increase harder to sell as an engineering achievement without an unchanged rerun beside it. Gunner Technology's position is that the baseline needs to earn your confidence before the challenger can earn your money.

Keep the model, repository revision, tasks, scoring rules and execution limits fixed while you measure the baseline. Record the harness version, instructions, available tools, reasoning setting and network access too. Start each attempt in a clean workspace so it cannot inherit a previous patch or transcript. The comparison needs a record another operator can reproduce.

Decide how you'll handle broken infrastructure before running the experiment. A missing dependency and a rejected patch tell you different things. Preserve both, state which attempts were scored and apply the same replacement policy to every candidate. Don't quietly give the preferred setup another try while leaving its rival's interrupted attempt as a failure.

Keep spending visible alongside success. Our post The Search Budget Can Pick Your Model Winner explains how unequal allowances can choose the apparent winner. Here the extra question is whether a winner appears when the setup hasn't changed at all. Budget fairness and repeatability answer different buying questions; you need both.

Look at individual tasks because matching totals can hide different failures, and a higher total can hide a serious regression. Pair the results by task: what passed before, what passes now and where the outcome changes on another attempt. Keep the patches and checker results so a reviewer can investigate the difference.

Suppose, hypothetically, a new harness starts passing export tasks but intermittently breaks access checks it used to handle. The total could improve while the route becomes unacceptable for your product. Separate work by consequence before declaring a winner. A gain in routine formatting doesn't buy permission to weaken authentication checks.

Write the promotion rule before inspecting candidate scores. Name the improvement that would justify switching, the regressions that would block it and how much uncertainty you're prepared to carry. If repeated comparisons still can't distinguish the candidate from the baseline, report the result as inconclusive. Buying another batch of runs should depend on whether the decision is valuable enough to resolve.

Yes, if it delivers a separately measured benefit without violating the route's acceptance requirements. Lower operating cost, fewer interrupted attempts or better evidence can justify a change. Name that benefit directly. You don't need to turn every useful improvement into a claim that the agent became smarter.

For example, test whether a tool now starts reliably in the restricted environment where you'll run it. Test whether interrupted work preserves enough state to resume safely. These are specific operating properties with observable outcomes. A benchmark score can miss them, so give them their own checks instead of assuming a larger task suite will answer every question.

Count the migration work too: changed instructions, repaired integrations, operator training and the route back if the upgrade fails. As Quality Is Part of the Agent Price argues, a cheap attempt can transfer costs to whoever repairs it. A harness swap should reduce the complete burden of accepted work enough to justify the disruption.

Start with a recurring workload in a repository you own. Save its baseline configuration and repeat the evaluation before changing the harness. Then test the candidate against the same work, with independent checking outside the builder's control. The agent proposing an upgrade shouldn't get to redefine what counts as success.

Have the factory retain the configuration, results, costs and failure evidence together. A person decides whether the measured benefit matters and whether the remaining uncertainty is acceptable. The machinery should run the comparisons and prepare that decision, rather than making someone reconstruct it from chat logs.

A disappointing result can be useful: you've avoided a migration that hasn't earned its cost. A repeatable improvement gives you a reason to move. Before approving the next harness rebuild, ask for the unchanged rerun. Your factory needs evidence that the change helped, and a new high score alone cannot tell you that.