You Are Benchmarking the Harness

A coding model never meets a repository by itself. Something chooses the files it sees, describes the tools it can use, decides when it may run them, preserves or drops its earlier work, and tells it when to stop. That surrounding machinery is the harness. Change it and you change the job the model is capable of doing.

This is why model comparisons get slippery so quickly. A strong result may come from better context retrieval, a more useful edit interface, a smarter retry loop, or a test command that exposes failure at the right moment. A weak result may come from the opposite. Calling all of that model performance hides the mechanism that produced the outcome.

The distinction matters outside a benchmark. You cannot buy a score and install it into your company. You can only operate a complete route: requirements enter, agents act through bounded tools, evidence comes back, and an authority decides whether the work moves. The engine matters. The vehicle decides whether it reaches the destination.

Every agent run has limits, whether you name them or not. There is only so much context, time, money, and permission available. The harness spends those resources. It can burn context rereading irrelevant files, waste tool calls rediscovering repository commands, or keep retrying a broken approach because no failure signal changes the route.

A good harness spends differently. It presents the smallest useful context, exposes stable verbs for common work, and brings back evidence the agent can act on. It remembers decisions that should survive the current session. It also stops. Unlimited retries do not turn a bad route into a reliable one; they turn uncertainty into a larger bill.

That means cost and capability cannot be separated cleanly. A cheaper model inside disciplined machinery may finish work that a stronger model fumbles inside a noisy loop. Our position is that the model will rarely be the lasting advantage. The advantage is the system that converts its capability into repeatable work without handing it unlimited authority.

A useful evaluation keeps the question honest. If you want to compare models, hold the harness still. If you want to compare harnesses, keep the model, repository, task, budget, and proof conditions still. If everything changes at once, the winning score gives you a result without a decision.

Then move beyond the clean task. Run the system against the conditions your software lives in: incomplete requirements, unfamiliar modules, failing tests, permission boundaries, generated files, and changes that need to be reversed. Measure what survives the route, not what looks convincing at the end of a transcript.

The final proof must sit outside the builder's control. An agent that chooses the task, the test, and the interpretation of the result is not being evaluated. It is presenting its own case. The gate should reproduce the work in conditions the harness cannot quietly soften after seeing what the agent produced.

Once the harness is visible, model choice gets less dramatic. Give every candidate the same input contract, tool boundaries, evidence requirements, and stop conditions. Promote a model when it improves the route. Replace it when another one does. The factory should retain its standards even when the engine changes.

Do the same with the harness itself. Version its prompts, tool definitions, context rules, budgets, and retry policy. Treat a change to any of them as a production change, because it is one. Run the same evaluation set before promotion and keep the failure artifacts. Otherwise the system can drift while the model name stays reassuringly constant.

This also makes improvement legible. When a failure becomes a regression test, the gain belongs to the factory instead of one person's memory. When a tool route gets tighter, every future agent inherits the correction. Capability that compounds in machinery is an asset. Capability trapped in a clever operator is a staffing dependency.

People still choose what deserves to be built, which risks are acceptable, and what evidence is enough. They should not spend their days manually rescuing context, repeating repository instructions, or watching an agent click through work the harness could govern. Those are repeatable operations, and repeatable operations belong in the machine.

Some jobs built around that supervision will disappear. That is not a side effect we should hide behind nicer language. Organizations that encode the work will need fewer people to carry process from one run to the next. The people who remain will own the destination, the boundaries, and the consequences rather than performing the loop by hand.

So stop asking which model won in isolation. Ask which complete system turns intent into software you can prove, at a cost and authority level you can defend. Benchmark the engine when you need to choose an engine. But when you are deciding how your company will build, benchmark the harness that has to bring the work home.