2026-11-20Gunner Technology
Count Every Model Behind the Result
Allow helper agents when testing the coding setup you plan to operate, and disable them when isolating a model's performance. Record which models actually ran either way. A result labeled with the lead model can hide a different crew doing part of the work.
What happened in the MAI Code 2 test?
The author of this MAI Code 2 comparison tested the model through GitHub Copilot CLI, a coding tool operated from the terminal, and left its default settings enabled. Those settings allowed subagents: helper agents that take on a portion of the lead agent's work. The article says Microsoft's reported benchmark disabled them.
The author's setup launched helpers in four runs, including a call to a different model. That tells us something about what ran. It does not establish whether the helpers improved the result, caused a failure or made any meaningful difference. The article reports no controlled comparison with helpers disabled.
The broader test found that fixing the requested behavior could still break existing behavior. Its tasks came from a single codebase, which limits how far the ranking travels. Read it as evidence about that tested setup. Gunner Technology's recommendation is to make the helper configuration explicit before using a result to choose your own setup.
When should helpers be switched off?
Switch helpers off when your question is what the selected model can do without delegated assistance. Enforce that setting in the harness, the software that gives the model its context, tools and execution loop. An instruction asking the lead agent to work alone leaves the experiment dependent on its compliance.
Hold the repository snapshot, task, tool access, reasoning setting and acceptance checks steady. Confirm in the execution record that no helper ran. If a candidate cannot operate under that restriction, say so and leave it outside that comparison.
Then run a separate trial with the helpers you intend to use. That trial answers whether the whole setup is useful. As our explanation of benchmarking the harness shows, changing the surrounding machinery changes what the result can support.
What belongs in the run record?
Keep the identity, assignment and usage of each helper beside the task it served. Record the lead model, helper models, harness version, delegation settings and any fallback that actually occurred. Preserve the service's reported model identifier; when that identifier is unavailable, mark it unknown.
The execution system should collect this record. Asking the lead agent to remember whom it called gives the experiment another unverified answer. Connect each helper session to the session that started it, including helpers that fail, time out or return work the lead agent discards.
Keep their charges attached too. Our guide to allocating agent costs to accepted work explains how failed attempts stay on the bill. A successful lead session should carry the cost of the help it used, even when the final patch contains none of the helper's edits.
How do you compare different helper setups?
Compare complete setups against the same tasks, acceptance requirements, spending ceiling and deadline. Let each use its declared helper configuration, then report that configuration with its results. This makes the comparison useful for choosing an operating route, meaning the sequence of agents, tools and checks that handles the work.
Count spending across the lead and its helpers. Measure elapsed time from the start of the job through acceptance, since parallel sessions can spend money together while the clock advances once. Keep incomplete and rejected attempts in the record. Our guide to comparing agents within a shared budget covers the spending boundary.
To find out whether delegation helped, repeat the task set with only the helper setting changed. Keep the acceptance checker outside the builders' control, and check existing behavior as well as the requested fix. Until that comparison exists, credit the complete setup for its result and leave the helper's contribution unresolved.
What if a helper cannot run?
Test what the setup does when a helper is unavailable before depending on that helper in unattended work. It might stop, retry, continue alone or choose another model. Each response changes the conditions under which the original result was earned.
In a trial environment, deny the helper route and observe what happens. Require any replacement to stay within the approved data destinations, tool permissions and total budget. Permission to delegate should never let the lead agent expand what the job is authorized to do.
Keep the failed call and the recovery attached to the same attempt. If the route needs a person's decision, it should stop with the missing dependency and the available choices. Hiding that intervention would credit the software with a decision a person supplied.
Keep the crew with the result
Gunner Technology recommends promoting a tested configuration together: lead model, permitted helpers, access boundaries, spending limit and independent checks. A helper change deserves another trial even when the lead model's name stays the same. Keep the configuration in your own repository so your team can inspect and replace it.
Start with a recurring task in a repository you own. Have the factory retain the execution record and run the checks. People decide what behavior is wanted and whether the returned work is right. When you read the result, you should be able to see the whole crew that earned it.
In response to The Promo Footnote: MAI Code 2 vs Opus 5.5 on Real Code by Emergent Minds | paddo.dev.
Quick answers
Does a different helper model invalidate a result?
No. It makes the result evidence about the combined setup, and the helper configuration needs to stay attached to that evidence.
Can a helper also perform the acceptance review?
An informal helper review can improve a draft. Acceptance still needs separate authority and checks the builders cannot weaken.
What should we do when helper usage is unavailable?
Mark the cost incomplete and preserve the known usage. Avoid a savings claim that depends on treating the missing portion as free.