Take the Winning Task Out

A winning task can hide how little you know about the rest of the workload. A benchmark is a set of jobs run under recorded conditions and judged against an acceptance standard. Its headline result depends on which jobs went in. Before buying capacity around that result, find out which jobs carried the decision.

In paddo's coding-model comparison, a particular change drove the difference between the leading candidates. The author also acknowledged that another codebase could put different work in that decisive slot. That's a useful limit on the result. It tells a buyer where to investigate next without pretending the comparison settles every repository's needs.

Gunner Technology's position is that a model should get more work like the jobs it has proved it can handle. If it handles a particular compatibility problem better, test more compatibility work. Don't turn that finding into authority over unrelated changes. The useful question is what the winning task reveals about the jobs you actually need done.

Count a task as a distinct request with its own starting state and acceptance conditions. Repeating a run tests whether the result recurs on that request; it doesn't show how the model handles different requests. Keep the job identity attached to every attempt so repetition can't masquerade as broader coverage.

Choose jobs from the queue you intend to automate. Include ordinary work, work that previously needed rescue, and changes with consequences that matter to your business. Separate routine interface edits, dependency updates and permission changes. Their frequency and failure costs differ, so give each category a visible place in the decision.

Decide that mix before looking at candidate results. Write down why each task belongs and preserve the repository snapshot, request and independent checks. As we explain in You Are Benchmarking the Harness, the surrounding tools and instructions affect the result too. Keep those conditions steady while investigating the influence of the tasks.

Recalculate your decision with each task omitted in turn, keeping all attempts on that task together. This is a sensitivity check: a way to see whether changing part of the evidence changes the conclusion. Use the same acceptance rules and cost accounting each time. Keep the full result beside every reduced result.

If dropping a job changes your choice, inspect what made that job different. Did it require finding a dependency across modules, preserving an obscure behavior or following an unusually precise instruction? A repeated failure may expose a real capability gap. It may also expose missing context or a faulty check. Investigate before assigning a cause.

Don't delete the awkward task to produce a tidier ranking. Add fresh work that exercises the same requirement, and check whether the difference persists. Keep that new work separate from the examples used to tune prompts or tools. Otherwise you may measure how well the system learned your benchmark instead of whether it can handle the next request.

Keep shared failures, abandoned runs and the full bill visible alongside the wins. When every candidate fails a job, their ranking cannot tell you how that job will get finished. It still needs a route: better intake, a different tool, a stronger check or a question for the person who owns the requirement.

Keep cost and elapsed time attached to each task category. A model may justify extra spending on a difficult repair while offering little benefit on ordinary edits. Our post on prompt caching in model comparisons covers another condition that can change the bill. Record it rather than letting a setup difference pass for a capability gain.

For permission changes, require the checks for unauthorized behavior to pass before considering savings elsewhere. A pile of inexpensive successes doesn't compensate for granting the wrong access. People decide that boundary in advance; the factory enforces it when candidates run. A weighted average should never silently override a release requirement.

Promote the model for the category of work its evidence supports. Record the task conditions, required checks, spending allowance and fallback route. When the advantage rests on a narrow capability, start there. You can expand its role after fresh work earns that expansion.

Keep the evaluation inside your own accounts, with access limited to the repository and tools it needs. The machinery should replay jobs, collect results and flag changes in the ranking. People should judge whether the task mix still represents the business and whether the observed failures are acceptable. Rebuilding a comparison spreadsheet for every model launch is repeatable work the factory can carry.

Start with a queue in a repository you own. Inspect the winning jobs as carefully as the failures. Then take each task out and ask whether you'd still make the same buying decision. If the answer changes, you've found the next experiment to run before expanding the model's role.