A Hidden Test Can Still Be Wrong

Keeping tasks private limits the opportunity to prepare specifically for those tasks. A benchmark is a collection of jobs used to compare systems under recorded conditions. Its answer key and grading rules decide which attempts count as successful. Those decisions need scrutiny even when the jobs stay out of public view.

Artificial Analysis's Intelligence Index update describes using private test sets or withheld answers while also revising ambiguous and erroneous knowledge questions. These are separate responsibilities: protecting an evaluation from exposure and checking whether its questions deserve to decide a result.

Gunner Technology recommends treating evaluation content as maintained software. Someone must own its correctness, approve changes and explain what each version measures. A private file can contain a bad requirement just as easily as a public file. Restricting who can read it doesn't repair the requirement.

An authorized reviewer who didn't build the candidate should be able to inspect the task, expected outcome and reason for that outcome. Independence means the reviewer can reject a faulty verdict without needing the builder's permission. It doesn't require publishing the exam to everyone who might take it.

For your own coding-agent evaluation, assign that responsibility before running candidates. Give the reviewer access inside your accounts, keep the tasks out of the builder's context and record who inspected them. Ask whether the instructions are sufficient, the starting environment works and the expected result follows from the request. Check the grading code too.

For an external leaderboard, ask the publisher how it reviews questions and resolves challenges. If it cannot disclose the tasks, it can still explain that process and the limits on its claims. Use results without an inspectable review process to decide what to investigate next. Don't make them the sole reason to replace a worker already handling your queue.

Compare the recorded attempt with the approved requirement and inspect why the grader rejected it. The grader is the mechanism that assigns the result. Its rejection is evidence to investigate, not a substitute for understanding what happened. Preserve the request, environment, output and grading details together.

Consider a hypothetical export task that asks for records in date order without saying which direction. The hidden answer expects newest first; the agent returns oldest first. The rejection exposes an incomplete instruction. It hasn't established that the agent cannot sort. Resolve the missing product decision before using that case to judge capability.

A different case may reveal a correct requirement and a broken grader, such as one that rejects an equivalent output format the task allows. Keep those causes distinct from an agent that ignores an explicit instruction. As Take the Winning Task Out explains, a particular task can carry a buying decision. Its verdict needs to survive examination.

Issue a new evaluation version and apply the correction consistently to every affected candidate. Keep the original results as history, with the reason they no longer support the current comparison. A score belongs to the task set, grading rules and execution conditions that produced it.

If only the grader was wrong and the preserved outputs contain everything needed, you may be able to grade those outputs again. If the instruction changes, run the affected tasks again under the corrected instruction. An old response to an ambiguous request doesn't tell you what the candidate would do with the ambiguity removed.

Keep unresolved cases visible and explain how they affect the conclusion. Don't remove a disputed failure only for the model you prefer. Test whether the choice changes with that case excluded consistently across candidates. If it does, hold the buying decision until you have better evidence or limit the trial to work the evidence supports.

Build a challenge route that preserves the evidence and sends the disputed requirement to its owner. The builder may flag a suspected test defect, but it mustn't rewrite the expected answer to pass. A separate reviewer decides whether the task, grader or implementation needs repair. The factory records and carries out that decision.

When resolving a challenge exposes a hidden task to the builder, record that exposure. Keep the corrected case for protecting known behavior, and use fresh work when you need independent evidence of improvement. Our post Your Hidden Tests Can Become Training Data covers that next problem. Fixing an answer key doesn't make an exposed question unseen again.

Start with a recurring queue in a repository you own. Audit the expected outcomes before comparing agents, then keep the challenge route available when results arrive. People decide what correct means and judge whether the returned work is right. The factory should preserve that meaning through every test and correction, rather than letting a secret answer become unquestionable authority.