AI Usage Is Not Factory Performance

AI tools produce an inviting trail of measurements. You can count active users, prompts, generated changes, agent sessions, and accepted suggestions. Those numbers answer a real question: is the organization touching the technology? They do not answer the question leaders actually care about, which is whether the organization can deliver better software with less human coordination.

The distinction matters because activity grows before the operating system around it does. A developer can run an agent all day and still spend the afternoon repairing its work. A team can generate more code while its review queue grows. A company can buy broad access while every useful run still depends on one experienced person packing the context, supervising the result, and cleaning up afterward.

None of that makes usage data worthless. It makes it an input. Treating it as the outcome rewards motion, and agents are exceptionally good at producing motion.

The useful unit is not a user with an AI license. It is a piece of work moving from a decided requirement to a result running under real conditions. Follow that route. Ask how much human intervention it needed, which gates rejected it, what it cost, whether it stayed fixed, and what the next run inherited from the experience.

That view exposes patterns a usage chart hides. Heavy activity paired with repeated rework points to weak specifications or weak proof. Low activity paired with reliable autonomous completion may be the healthier system. A model that writes fewer lines but survives review and production can be more valuable than one that fills the repository and creates a larger judgment queue.

Measure accepted outcomes at the boundary that matters. For a defect, that means the failure was reproduced, the correction survived independent checks, and the behavior held after release. For a feature, it means the promised behavior exists and the evidence matches the promise. Code volume is incidental to both.

A single adoption number also collapses work that should never be compared. A small documentation correction, a dependency migration, a new payment path, and an ambiguous product decision place different demands on an agent. Mix them together and the average tells you almost nothing about where autonomy is real.

Classify work by risk, ambiguity, and the proof available. Then compare routes within those classes. Which tasks complete without a person stepping into the middle? Which need a stronger model, more context, or a different verifier? Which should stop before implementation because the requirement cannot yet support an objective gate? Those are factory decisions.

The segmentation must drive machinery. If a route performs reliably, expand its authority. If failures cluster around missing context, repair the context stage. If a cheap worker succeeds under strong checks, keep it there. A report that never changes routing, permissions, budgets, or proof is only a description of yesterday.

Most organizations will discover that the first constraint is not model capability. It is human availability. Someone has to clarify the request, select the context, watch the run, review the change, approve an exception, or translate a production failure back into the next prompt. The agent looks fast because the waiting is recorded somewhere else.

Put that waiting back into the measurement. Count interventions, handoffs, and elapsed time between machine steps. Separate judgment that genuinely needs a person from repeatable inspection that has simply not been encoded yet. The first category belongs at an explicit authority gate. The second belongs on the factory backlog.

This is the uncomfortable consequence of measuring honestly. When a team turns its recurring review decisions into specifications and executable checks, it removes work that people perform today. Our position is that those jobs will shrink. The durable human contribution is choosing the destination and deciding which consequences are acceptable, not remaining the permanent adapter between fast agents and an unfinished system.

A good dashboard should change the factory. It should show where work stalls, which evidence predicts survival, where costs outrun value, and which interventions repeat often enough to become controls. Every important signal should have an owner and a possible response.

That means pairing usage with accepted work, independent proof, production behavior, spend, and retained learning. It means tracking a result across the whole loop instead of celebrating the moment an agent emits code. It also means resisting the easy comparison between people. The goal is not to rank who prompts the most. The goal is to remove prompting, chasing, and routine checking from the route altogether.

Adoption tells you the tools are present. Performance tells you the factory can take more authority without creating hidden repair work. Measure that difference, and the data becomes a control surface. Miss it, and you will optimize the part of software delivery agents already make cheapest: producing more output.