The Agent Needs a Real System to Test

Coding agents make the feedback loop look complete. They change the code, run the suite, inspect the result, and keep working until the checks turn green. That is real progress. It is also only as reliable as the world those checks describe.

Most useful software does not live alone. It talks to services, databases, queues, identity systems, and third-party interfaces. Replace all of that with local stand-ins and the agent proves compatibility with the stand-ins. A person may remember which mock is stale or which happy path hides a production wrinkle. The agent sees a passed check and a finish line.

That is not a model problem. It is a verification design problem. You gave the worker a simplified world, then asked it to make a production claim. More confident reasoning cannot repair evidence collected from the wrong conditions.

Intent files, specifications, plans, skills, and review policies give an agent durable context. They tell it what the organization decided, how the work should move, and which rules it must obey. They do not tell it what an upstream service returns today or which schema is actually running in staging.

The article behind this dispatch makes that gap concrete: an agent-native lifecycle can preserve a strong chain of repository artifacts while still leaving the worker blind to the live environment around its code. That blind spot lands hardest at verification, because the check can be mechanically correct and operationally irrelevant at the same time.

A software factory needs both kinds of truth. The written record carries intent. A production-shaped environment carries current behavior. Verification is the controlled meeting between them, where the change has to survive the system it will actually join.

Pointing an agent at shared staging is not the answer by itself. Several workers can collide through the same queue, database rows, or service traffic. One run changes the conditions beneath another. Engineers lose their environment, tests become nondeterministic, and nobody can say which worker produced the evidence.

The useful pattern is shared realism with isolated effects. Let the change reach production-shaped dependencies, but give each run its own traffic boundary, data branch, queue slice, and identity. The worker sees the real contracts without inheriting permission to disturb every other session using them.

That isolation is factory infrastructure, not developer convenience. It makes parallel work attributable. It gives a failed check a reproducible owner and a successful check a defensible scope. Without it, concurrency multiplies uncertainty faster than it multiplies output.

The builder should not get to choose the only world in which its work is judged. If the same agent writes the code, defines the stand-ins, selects the data, and explains the result, the whole loop shares one assumption. A polished report does not create independence.

Put the environment contract outside the worker. The factory decides which dependencies are realistic, which effects are isolated, which secrets are available, and what evidence must come back. The agent can explore and repair inside that boundary. It cannot lower the boundary because the result is inconvenient.

Then make the evidence portable. Record the environment version, routed traffic, relevant data state, checks performed, and observed behavior. Another verifier should be able to reproduce the claim without trusting the builder's narration. That is how a test result becomes operating proof instead of a green mark attached to a session.

Our position is that verification infrastructure will become the limiting factor in agent-run delivery. Code generation will keep getting cheaper. The ability to run many changes against realistic systems, without letting those changes interfere with one another, will decide how much of that output can safely move.

Start with one service boundary. Identify the fake your team trusts mostly from habit. Give one agent run a controlled route to the production-shaped dependency, isolate its effects, and compare what the stronger environment reveals. Turn every mismatch into a permanent acceptance condition.

The factory does not become real when an agent can write the code. It becomes real when the work can face the system, produce evidence under governed conditions, and survive a verifier that did not build it. Give the agent a real system to test. Give the system a boundary the agent cannot erase.