2026-11-16Gunner Technology
The Median Isn't Your Agent Budget
Median cost per pass cannot tell you what your coding agents will spend across a queue. You need the complete cost of the attempted work and the outcomes that passed, including the bill for failures. A useful benchmark can point you toward a cheaper harness without giving you a budget you can operate against.
What does a harness cost benchmark actually tell you?
A harness cost benchmark tells you how the tested setups performed under the conditions recorded. The harness is the software around a model that supplies instructions, runs tools and manages the work. Holding the model steady helps expose differences in that surrounding machinery.
Developers Digest's explanation of a coding-agent harness benchmark describes substantial differences in reported cost despite closer pass rates, and labels its cost column median cost per pass. It also flags limits involving the tasks, model and tested versions. That is a reason to investigate harness economics, with those limits attached.
The article also describes another leaderboard with a different ordering. Its useful lesson is that a ranking belongs to a workload and configuration. It does not establish which harness will be cheapest in your repository, or how much money your next queue requires.
What can median cost per pass hide?
A median can hide how expensive the observations above the middle become. The median is the middle value in an ordered set, with the middle pair averaged when needed. It describes the center; it does not preserve the total. Expensive outliers can grow while the median stays put.
First ask what an observation means. Is the reported median taken across tasks, repeated batches or configurations? Then ask how each cost-per-pass value was calculated. Did its numerator include failed attempts, timeouts and retries? A column label alone cannot settle those questions. Do not assume the benchmark excluded failures, and do not assume it included everything your factory pays for.
Even a carefully calculated median cannot reconstruct the costs that produced it. Nor can you combine a median cost and an overall pass rate to recover a missing spend total. Ask for the underlying records. The summary may be completely accurate while still being insufficient for the purchasing decision you want it to support.
How do you turn the result into a budget comparison?
Build the budget comparison from total measured cost and accepted outcomes over the same bounded batch. Define the batch before it runs, keep unfinished and rejected work in its record, and state where measurement stops. A benchmark pass and a production acceptance decision are different endpoints; name the one you measured.
For each harness, retain the task, starting code, configuration, attempt history, usage and verdict. Include the resources inside your stated boundary: model calls, tool execution, checks and any intervention. Keep unpriced human time visible as time rather than silently assigning it no cost. Our guide to allocating agent costs to accepted work explains how to keep those events attached to the job.
Divide the batch's total included cost by its accepted outcomes to calculate the batch cost per accepted outcome. Keep that ratio beside total spend, acceptance and the distribution of individual costs. If nothing passes, report the spend and failed outcome directly; there is no finite cost per accepted outcome to celebrate.
When should the expensive runs change your choice?
Expensive runs should change your choice when they expose a recurring failure your operating limits cannot absorb. Imagine a harness that handles ordinary edits cheaply but keeps retrying an integration it cannot finish. That is an illustrative failure mode, not a reported result from the article. Its consequence depends on whether your queue contains that work and whether the harness stops.
Inspect those runs separately. Did the agent lose context, repeat a failed tool call, exhaust a budget or wait for a missing decision? A rare costly attempt might be justified by a valuable accepted result. Repeated spending without stronger evidence calls for a different route or an earlier stop. The median cannot make that judgment for you.
Compare repeated batches before treating a difference as dependable. Rerun the unchanged baseline so ordinary variation does not become a reason to replace the machinery. Keep permissions and acceptance checks steady; a cheaper route that quietly drops required proof has changed the product being compared.
What should you ask for before choosing a harness?
Ask for the metric definition, the run records and a limited trial on work you actually intend to send. A vendor should be able to explain what its cost includes, what counts as passing, which runs were excluded and why. An unexplained exclusion belongs in the buying decision, even when the headline looks attractive.
Then connect the trial to your expected queue. Different proportions of routine fixes and difficult integrations can change the bill without changing either harness. Use the records to forecast costs by work class, and put spending limits where they can stop new attempts. Retain a route back if the expected saving disappears.
Gunner Technology recommends keeping the median as a description of typical measured behavior, with complete costs beside it. People choose the work worth funding and judge whether the result earns acceptance. The factory should collect the bill, preserve failures and stop spending at the agreed boundary. Buy against that record.
In response to Coding Agent Harness Benchmark: Same Model, 17.5x Cost Gap by Developers Digest.
Quick answers
Is the median still useful on an operating dashboard?
Yes. Use it to track typical measured cost under a consistent definition, with total spend, acceptance and expensive runs visible beside it.
How should a flat subscription appear in the trial?
Record the subscription charge and the allocation rule separately from usage. State any capacity limits so unused allowance does not look like unlimited free execution.
Should a run that hits its budget be deleted from the results?
No. Keep its spend and its unresolved or failed outcome in the batch. The budget stop is part of how the harness operates.