A Testing Tool Is Not a Testing Strategy

An agent can install the requested library, write valid test code, run it, and produce a clean result. Every visible step happened. That does not mean the test reached the behavior most likely to break. A property test that mostly generates rejected inputs is activity, not pressure. A proof about an easy invariant is still silent about the dangerous one.

This is the trap in naming a technique and treating the name as a control. The agent optimizes for the instruction it can see. If the instruction says to use a tool, successful tool use becomes the target. The factory needs correctness, but the route never defined which failures matter or how the test could expose them.

More tests do not repair that mismatch. They can make it worse by surrounding the implementation with green checks that all share the same blind spot. A wall of passing tests is persuasive evidence only when those tests had a credible way to fail.

Verification should begin by asking where this implementation is fragile. Which transformations reverse order? Which state transitions depend on timing? Which boundaries accept malformed data? Which assumptions are copied into both the code and its tests? Those questions choose the technique. The technique cannot choose the questions for you.

A factory should turn that risk map into a verification plan before it asks for test code. Name the behaviors that deserve pressure, the input regions that should be overrepresented, and the consequences that require a stop. Then route each risk to the cheapest method that can expose it. Sometimes that is a table-driven unit test. Sometimes it is structured generation, mutation, differential comparison, or a formal model.

The fashionable method does not get a vote. If one sharp example can kill the bug, use it. If a broad input space hides interactions no one can enumerate, generate structured cases around those interactions. The job is to find the failure, not to demonstrate membership in a testing school.

Agent-written tests often inherit the implementation's mistake because both came from the same interpretation. The code produces an answer, the test records that answer, and the suite congratulates the system for agreeing with itself. Running the loop more often only hardens the error.

Break that symmetry. Derive expected behavior from a separate specification, a known-good implementation, a protocol corpus, or invariants written before the builder sees its own output. Where possible, run the evidence in a different environment and under inputs the builder did not select. Independence is not ceremony. It is what gives disagreement a chance to appear.

This is also why asking the implementing agent to judge its own coverage is weak. It knows what it meant to build, so it naturally explains the code in the friendliest possible terms. A verification stage should arrive with a different objective: find the cheapest counterexample to the claim that this work is ready.

Good verification cannot depend on a person remembering to write a clever prompt on every change. Encode the route. A parser change should trigger malformed structure generation and boundary cases. A concurrent state machine should trigger schedule variation and invariant checks. A compatibility layer should compare behavior against the system it claims to match.

Then test the tests. Deliberately damage the implementation and see whether the suite objects. Seed known counterexamples. Track which risk classes have an oracle and which still depend on human inspection. A test suite that survives obviously wrong code is not incomplete documentation. It is a broken gate.

The factory should also notice waste. If a method adds cost without reaching new behavior, remove it or change the generator. If thousands of cases collapse into the same rejection path, steer the distribution. Verification is production machinery, and machinery that spins without applying pressure needs maintenance.

People still decide what failure would be unacceptable. They identify the business invariant, the security boundary, and the compatibility promise that the software cannot negotiate away. That judgment is valuable because it sets the target. Repeating test commands and sorting familiar failures is execution work, and agents will absorb it.

That shift will eliminate testing jobs built around manual repetition. It will also make shallow automation painfully visible. A team cannot claim quality because an agent generated a large suite or invoked an impressive tool. The only useful question is whether the verification system finds consequential mistakes before customers do.

Our position is simple: testing technique belongs inside the factory, but technique alone is not the factory. Define the risks, separate the oracle, force the evidence to attack the claim, and feed every escaped failure back into the route. Otherwise you have automated the appearance of caution while the dangerous behavior passes untouched.