Your Hidden Tests Can Become Training Data

Hidden tests lose independence when their feedback starts steering the changes they will judge. A holdout is a set of tasks kept out of development so it can check performance on work the builder hasn't practiced. Keeping its files secret doesn't keep its results out of the development loop.

An agent harness is the software around a model: its instructions, tools, memory and execution rules. If you change that harness after each evaluation score, the score helps choose the next version. Dwork and colleagues' research on adaptive data analysis explains how repeatedly adapting to a holdout can overfit that holdout. The system can become better at this exam without earning the broader claim you want to make.

That adds a requirement to The Harness Must Not Grade Itself. Separating the builder from the evaluator protects who controls the verdict. You also have to track which verdicts have already influenced the builder. A locked evaluator can still become a familiar opponent.

Keep tests that protect behavior you still require, even after agents have learned from them. A regression test checks that a previously working behavior hasn't broken. Familiarity doesn't make that job disappear. A permission boundary must keep holding after the worker knows exactly how you test it.

Gunner Technology recommends recording what each evaluation set is allowed to establish. Development cases help agents diagnose and improve. Regression cases protect known requirements. Fresh admission cases help decide whether a finished candidate deserves to replace the current harness. Admission means accepting that candidate into the machinery future jobs will use.

A case can move between those jobs. Once a failed admission case becomes a repair example, retain it as a regression check. Record that transition instead of continuing to present the repaired case as unseen evidence. The test still has value; the claim it supports has changed.

Freeze the candidate before choosing the fresh tasks that will judge it. Record the harness version, the model, the environment and the acceptance criteria. Have a separate evaluation route select representative work, then run the candidate and the current harness under comparable conditions.

The Self-Evolving Agent Harness preprint makes this ordering explicit in its conditional admission analysis: candidates are fixed before fresh independent validation tasks are drawn. Changes informed by that validation need fresh validation again. That is an assumption behind the analysis, not a guarantee that every self-modifying agent improves.

For a practical evaluation, prepare tasks from the kinds of work the proposed harness will actually receive. Include cases where the right action is to stop and ask. Have the evaluation owner check that the expected outcomes are correct and that private task details haven't entered the candidate's context. Rewording a familiar task doesn't establish independence.

Keep the current harness, preserve the failure and return the candidate to development. Once the repair uses that result, require another independent evaluation before promotion. Don't rerun the same exam until a passing score turns the development history into a success story.

Budget for that path before starting the search. Reserve capacity for final evaluation, and cap the attempts that can consume it. If the factory cannot afford fresh evidence, it can retain the working version and report that the candidate remains unproven. Running out of evaluation capacity doesn't lower the acceptance standard.

Read the failures by task type as well as in aggregate. A candidate that handles routine edits better may handle recovery worse. Decide which regressions are unacceptable before looking at the results. People own that tradeoff; the machinery should preserve the cases, apply the agreed criteria and make rejection routine.

Assign someone responsibility for keeping the evaluation relevant. That person decides which work it represents and what evidence would justify replacing the current harness. The factory should record which candidate saw which feedback, select approved tasks and preserve the results without requiring someone to carry that history between runs.

Review the evaluation when the actual workload changes. A harness accepted for documentation work hasn't thereby earned database migrations. New tools, different repositories and changed acceptance criteria deserve a new decision about coverage. Yesterday's evidence can remain accurate while no longer answering today's operating question.

Start with a recurring queue in a repository you own. Keep its useful regression tests, separate its development examples and arrange fresh admission work before the next harness experiment. As we argue in Your Agents Need a Learning Loop, experience should change the machinery. The evaluation process needs to remember what that machinery has already learned.