One Score Cannot Route a Factory

An aggregate score is useful because it compresses a complicated evaluation into something people can compare. That same compression is also the trap. Software work is not one task repeated at different difficulty levels. Planning across an unfamiliar system, repairing a contained defect, operating tools, reviewing a permission change, and explaining a failure demand different behavior.

One model may be excellent when the destination is clear and the tools are constrained. Another may be better when the work begins with ambiguity. A third may be cheap enough to run on every mechanical transformation even if it loses the headline contest. Fold those differences into one number and you get a ranking. You do not get an operating policy.

The score says what happened across somebody's chosen mixture of tests. Your factory has to decide what should happen on this task, under your constraints, with consequences you actually own.

Model selection starts after the work has a shape. What artifact is entering the station? What transformation must happen? Which tools may the worker use? What evidence must leave with the result? How expensive is a failed attempt, and where does authority stop? Without those answers, choosing a model is shopping before you know what you are buying.

A factory can make the choice concrete. Route a bounded refactor to the least expensive worker that repeatedly clears the checks. Send architectural planning to a worker that handles long dependencies well. Give consequential changes narrower tools, a stricter budget, and an independent verifier. Escalate when the first route fails for a known reason, not when somebody gets impatient with the answer.

This is not loyalty to a model family. It is a job description with an acceptance test. The worker earns the station by producing evidence under the conditions that station requires.

The strongest available model is an easy default because it postpones the routing decision. It can also turn every small transformation into premium work. At factory scale, that is not caution. It is a missing control that happens to arrive as a bill.

Price alone is not the answer either. A cheap attempt that fails repeatedly can cost more than one expensive attempt, especially when a person has to diagnose the wreckage. Measure the cost of accepted work: the run, the retries, the verification, the human decisions, and the defects that return after release. A low token bill attached to rejected output is not efficiency.

Budget should change behavior mechanically. A routine route can have a tight ceiling and a known fallback. A high-consequence decision can earn more reasoning and stronger proof. When consumption crosses the limit, the factory stops, narrows the task, or escalates. It does not keep spending because the model sounds close.

Public evaluations age fast. Models change, prices move, tools improve, and yesterday's difficult behavior becomes tomorrow's default. If every release forces your team to debate the market again, model churn has become a manual production dependency.

Keep a small set of preserved tasks from your own work instead. Remove sensitive data, hold inputs and authority steady, and judge candidates with the same gates the real station uses. Record what passed, what failed, what it cost, and which failure reached production anyway. Then let new models compete for routes without rewriting the factory around them.

The important evidence is not a polished answer or a broad reputation. It is whether a fresh worker can take the same bounded package, operate inside the same authority, and produce a result that survives proof it did not control.

Our position is that model choice will keep mattering, but choosing models by headline rank will matter less. The durable advantage is the machinery that can describe work, route it, limit it, test it, learn from failure, and replace the worker without losing the process.

That machinery also changes the human job. People choose which consequences are acceptable, define what counts as proof, and decide when a result deserves authority. They should not spend their day comparing release announcements, hand-carrying context between tools, or remembering which model worked last month. Once that choice becomes repeatable, it belongs in the factory.

Use the score as a signal. Test the candidate at a real station. Promote it when the evidence holds, demote it when the route stops paying, and keep the business interface unchanged. A leaderboard can tell you which worker deserves an audition. Only your factory can decide who gets the job.