The Search Budget Can Pick Your Model Winner

You're comparing a complete route unless you hold everything around the model steady. The route includes its instructions, search tools, available documents and permission to keep investigating. A benchmark is a repeatable test against defined tasks. Its score belongs to the setup that ran those tasks, including the limits imposed on it.

Search makes this especially visible. OpenRouter's BrowseComp methodology holds the model fixed while varying the search engine, request format and maximum search budget. That experiment asks what the search configuration contributes. Swap those settings along with the model and you lose the ability to say which change helped.

For a software factory, that distinction changes where you spend money. If better retrieval solves the problem, buying stronger reasoning may pay for the wrong repair. If the agent already has the evidence but can't use it correctly, another search allowance won't settle the decision. Diagnose the missing capability before changing the whole route.

The same budget means a shared spending ceiling for the whole attempt, with a deadline and tool limits recorded separately. Equal token allowances don't establish equal cost across differently priced systems. Tokens are the units a model processes; a tool call is a request to another service. Your test needs to account for both.

Record model consumption, search charges and any paid processing the route triggers. Keep cache conditions explicit too: reusable input can change the bill, as we explain in Your Cache Is Part of the Model Comparison. When a charge is unavailable, mark the run's cost incomplete. Missing usage isn't a saving.

A common spending ceiling answers a buying question: what useful result can this route deliver within what we're prepared to pay? A common search limit answers a different question: what can each configuration do with the same permitted search depth? Run the experiment that matches the decision, and name the difference in the result.

Change a defined part of the setup while keeping the tasks and grading steady. Start by holding the model and its reasoning setting fixed while you vary retrieval. Reasoning settings control how much internal work a model is asked to do before answering. Then hold the search setup fixed while you compare candidate models.

Keep a separate experiment for the complete production route, where each candidate can use its appropriate configuration inside the shared spending ceiling. A model that needs different instructions can still be a useful purchase. Record those adaptations so the result supports a claim about that route rather than a claim about the model alone.

Test the same task set, preserve failures and repeat promising comparisons on fresh work. Don't let a candidate choose which questions count after seeing its answers. The controls we discuss in Repository Search Belongs in the Agent Budget belong in the evaluation too: retrieval needs a purpose, an allowance and a stopping condition.

A useful answer meets the acceptance standard you set before the run, with evidence another checker can inspect. For research work, define what must be correct, which claims need supporting sources and what uncertainty must remain visible. Returning a fluent report is an output event. Acceptance requires judging its substance.

Keep benchmark success in its proper scope. The BrowseComp research paper describes difficult browsing questions with short, verifiable answers, and explicitly leaves out challenges such as long answers and ambiguous requests. A strong result there can justify further testing. It doesn't establish that an agent can write a sound recommendation for your business.

Count abandoned attempts and exhausted budgets in the result. Preserve whatever they found, but don't quietly restart them with a larger allowance and report only the successful continuation. If a person supplies the missing insight, record that intervention. Otherwise the experiment has purchased human judgment and credited the agent.

Gunner Technology's position is that an evaluation should end with a routing rule you can test again. Name the workload the candidate passed, the spending boundary it stayed inside and the conditions that would send the work elsewhere. A general declaration that a model is best gives the factory little it can enforce.

Start with a recurring research task that feeds a product decision in a repository you own. Keep independent checking outside the research worker's allowance so it can't spend the proof budget on another search. People decide which question matters and judge whether the resulting evidence is enough; the machinery should run the comparison and retain the record.

Before buying a new default, ask to see the comparison with its limits attached. If the apparent advantage disappears when the spending boundary is shared, you've learned something useful. You can still choose the more expensive route when the result earns it. Now you know what you're paying for.