--- title: "Your First AI Coding Week Is Not the Verdict" description: "Measure an AI coding pilot by accepted work, rework and intervention over repeated runs, not by first-week speed or tool usage." author: "Gunner Technology" published: "2026-11-11" url: "https://www.gunnertech.com/blog/measure-an-ai-coding-pilot-during-adoption" --- # Your First AI Coding Week Is Not the Verdict Measure an AI coding pilot by the work that passes acceptance, the rework it creates and the help it needs over repeated runs. Do not declare victory from a fast demo or failure from the first awkward week. You are measuring a new operating route and the team's ability to use it, not merely a tool's output. ## What should an AI coding pilot measure? An AI coding pilot should measure accepted work across the complete route. Pick a recurring class of work, define what correct means, and follow each item from intake through review, release and any repair. Generated code, agent sessions and opened pull requests are activity. They become useful only when the result clears the checks and judgment required for that work. Record elapsed time, attempts, rejected changes, human intervention and production reversals for every item you attempt, including work that is abandoned or permanently rejected. Track whether each item was accepted as a separate field, so the items that never reach acceptance stay in the comparison. Human intervention means a person had to supply missing context, correct a decision, rescue the tools or take over the work. That is not automatically a failure. It tells you which part of the route is still manual and what the factory must learn next. Keep tool usage out of the headline. A rising prompt count may mean people are learning, or it may mean they are stuck. [AI usage is not factory performance](https://www.gunnertech.com/blog/ai-usage-is-not-factory-performance) because activity cannot tell you whether the software survived acceptance. Count the outcome first, then use the activity to explain it. ## How long should you wait before judging the pilot? Wait until the same route has run enough times to expose both learning and repetition. A single task is a demonstration. A useful pilot includes comparable work attempted under the same acceptance boundary, with each run preserving what changed in the instructions, tools and checks. Do not hide the early runs. The awkward first attempts show the cost of adoption, and the later attempts show whether that cost is falling. If the team needs less rescue because the route gained better context and clearer checks, that improvement belongs to the operating system around the agent. If the same confusion returns, more time alone will not fix it. Use ranges rather than inventing a clean productivity percentage from a small sample. Compare the pilot with the way the same class of work moved before, and [rerun the baseline before changing the harness](https://www.gunnertech.com/blog/rerun-the-baseline-before-changing-the-harness). Otherwise a different task mix, a broken check or a generous reviewer can look like progress. ## How do you separate a learning curve from a bad system? Separate them by turning every intervention into a named cause. Missing repository context, unclear acceptance, unsafe permission, tool failure and poor judgment are different failures. Put each one on the work item and ask whether the next run encountered it again. A learning curve improves when the lesson becomes reusable. The team stops rewriting the same prompt because the rule moves into the specification. Review stops rediscovering the same risk because a check now catches it. The agent stops waiting for a routine decision because authority is attached to that class of work. If progress depends on one expert remembering how to steer every session, the pilot has trained an operator, not built a route. This is why [AI adoption is not an operating model](https://www.gunnertech.com/blog/adoption-is-not-an-operating-model). Familiarity can make a manual loop faster, but a factory needs the result encoded in context, permissions, proof and recovery. The test is not whether your best user can make the agent work. It is whether the next comparable item inherits what that user learned. ## When is the pilot ready to expand? Expand when accepted outcomes become repeatable and the remaining interventions have owners. The route should state which work it accepts, what evidence it requires, where it stops and who can make the decisions it cannot. A strong result in one class of work is permission to widen one boundary, not proof that every team and repository should switch at once. Stop or redesign the route when retries stay high, reviewers cannot trust the evidence, or production repairs erase the apparent speed. Do not rescue the pilot by lowering acceptance. That trades a visible adoption problem for an invisible quality problem and guarantees that somebody downstream pays for the celebration. Gunner Technology's position is direct: the learning curve belongs in the pilot, but it does not excuse a route that never learns. Preserve the early cost, make each failure change the machinery, and judge the pilot by accepted work that becomes easier to repeat. That is how adoption turns into capacity instead of another dashboard full of enthusiasm. ## Quick answers ### Should a pilot include people who are new to AI coding tools? Yes, if they represent the people who will use the route. Keep their results distinct from experienced users so the adoption cost stays visible. ### What if the pilot has no clean historical baseline? Start with a narrow work class and record the current route before changing it. Use ranges and visible assumptions until repeated runs create a stronger comparison. ### Can a pilot succeed even when it needs human review? Yes. Human judgment belongs at consequential decisions; the problem is repeated manual rescue that the route never converts into a reusable control.