2026-11-18Gunner Technology
Make the Agent Prove It Found the Rules
Test whether a coding agent read the rules by giving it a small task whose correct result depends on the right scoped instruction, then check the result independently. Do not accept a confident summary as proof. An instruction file only governs the work if the agent can find it before acting and a separate check can catch the miss.
What are you actually testing?
You are testing instruction discovery, not whether the model can repeat a policy after you paste it into the prompt. Instruction discovery is the process that finds the rules which apply to a task, resolves their scope and puts them into the agent's working context before it changes anything.
That distinction matters because repositories rarely have one flat rulebook. A root instruction can apply everywhere while a more specific file governs one package, generated directory or deployment path. The agent has to start in the right place, walk the applicable hierarchy and understand which rule wins when scopes overlap.
Gunner Technology's position is that this belongs at intake. The factory should assemble the task and its governing context together. If every worker has to remember how to hunt for policy, you have a ritual, not a control. A prompt is a manual build step when a person must reconstruct that context for every run.
How do you design a useful instruction probe?
Use a small task with an observable fork: the general convention would produce one valid-looking result, but the applicable local rule requires another. The job should be harmless, quick to reset and specific enough that a checker can identify which path the agent followed.
For example, a fixture directory might require a local naming pattern or a generated file might forbid hand edits and require a repository command instead. The point is not to trick the agent with trivia. It is to prove that a rule located outside the immediate request changes the action the worker takes.
Run probes from the entry points your team actually uses. Starting in the repository root, opening a nested package and resuming an existing session can expose different discovery paths. Record the repository revision, working directory, agent version and task text. Otherwise a passing rerun tells you very little about what changed.
How do you prove the right rule was loaded?
Prove it with two pieces of evidence: the agent's action satisfies the scoped rule, and the factory records which instruction sources it supplied. The first shows behavior. The second makes a failure diagnosable. Neither is strong enough alone.
A correct edit can happen by chance or because the model already prefers the same convention. Add a second probe whose expected behavior is unusual but still reasonable, and vary the task without changing the rule. You are looking for consistent rule-governed behavior, not a lucky match.
Keep the checker beyond the builder's authority. A deterministic check is code that returns the same result for the same input; use one when the rule can be expressed mechanically. For judgment calls, have a separate reviewer inspect the artifact and the applicable rule set. The proof must outrank the builder, especially when the builder can write a persuasive explanation of its own miss.
What should happen when instruction discovery fails?
Stop the affected route and fix the discovery mechanism before adding louder prose. A missed rule may come from the wrong working directory, an unsupported filename, a scope resolver that stopped too early or context assembly that dropped the file. Name the broken link before changing the policy itself.
Then turn the probe into a permanent acceptance test for the harness, meaning the software that supplies context and tools to the model. Run it when the agent, wrapper, repository layout or instruction-loading code changes. The test should fail if the required rule is absent, not merely warn after the work is complete.
Do not make every instruction executable just to compensate for unreliable loading. Some rules describe judgment that code cannot settle. But when an instruction can be enforced in code, move that part into the gate. Let prose explain the decision and let machinery refuse the violation.
Buy the discovery path, not the demo
Before you trust a coding agent with unattended work, test how it finds repository rules from the exact entry points you will operate. Ask for the loaded-rule record, run scoped probes and verify the result outside the agent. A demo that follows a rule placed in the prompt proves only that the model can read the prompt.
Start with one repository and a handful of rules that change real decisions. People still decide what those rules mean and whether the result is right. The factory carries them to every worker, checks the outcome and stops when the chain breaks. That is how written policy becomes operating behavior instead of useful advice the agent may never see.
Quick answers
Should the agent list every instruction file before working?
Not necessarily. The factory should retain that record, but making the agent narrate it on every task adds noise without proving the work complied.
How many instruction probes do we need?
Start with one probe for each discovery boundary you depend on, such as the repository root and a nested package. Add a regression probe whenever a real miss reveals a new path.
Can a reviewer replace instruction-discovery tests?
No. A reviewer may catch the resulting mistake, but the same missing context can affect planning, tool use and changes that look reasonable in isolation.