2026-11-20Gunner Technology

Audit the Factory, Not the Prompt

Audit an AI coding repository by checking the controls that turn agent output into accepted software, not by grading the prompt file. Tie every gap to evidence, name the consequence and rerun the same checks after remediation. A long instruction document can look responsible while the repository still lets an agent skip testing, leak a secret or ship without an owner.

What should an AI coding repository audit check?

Check whether the repository can define the work, constrain the builder, test the result, protect access, release deliberately and recover when production disagrees. Those are control families, not a universal checklist. The exact requirements depend on what the software can change and what happens when it is wrong.
Start with the path a real change takes. Find where requirements enter, which local rules the agent receives, what tools it may call, which checks can reject its work and who has authority to approve a release. Then follow one failed change. If nobody can tell where it stops or what evidence travels with it, the audit has already found a gap.
Do not award points for file names. A security policy that no command enforces is context, not a gate. A test command that agents never run is an option, not a control. Our guide to enforcing agent instructions in code draws the useful line: prose explains judgment and intent; machinery blocks violations a machine can recognize.

How do you prove a control actually works?

Prove a control by attempting the violation it claims to stop. Put a known formatting error through the required command. Introduce a harmless fake secret pattern in an isolated fixture. Try to bypass a protected release route. The evidence is the refusal, including the command, the failing condition and the point where the work stopped.
Test the allowed path too. A guard that blocks every change is not safe; it is broken. Each control needs at least one rejection case and one permitted case, with both running under the same route an agent uses. Keep those cases small enough to rerun whenever the control changes.
The checker cannot take the builder's word for it. Capture exit status and artifacts outside the agent's narrative, and keep the acceptance condition out of the builder's write access when the consequence demands it. The proof must outrank the builder, or the repository has only automated self-approval.

What belongs in the audit report?

Report each gap as a missing decision or missing refusal, not as a vague maturity score. Name the affected route, show the evidence, explain the consequence and identify the owner who can accept or repair it. A finding such as ‘no secret scanning’ becomes useful when it says which paths can reach release without a scan and which command proves the gap.
Separate facts from recommendations. ‘The release job does not run the test suite’ is observable. ‘Add a required test job before deployment’ is a proposed repair. Keeping them separate lets the owner disagree with the remedy without erasing the evidence.
Avoid invented certification language. Your readiness label is an internal operating decision unless a real standard defines it. Publish the criteria and their versions so teams can see what passed, what failed and what was not examined. A single percentage hides whether the missing item is a style rule or a credential boundary.

Which audit gaps should you fix first?

Fix the gaps that let work cross a consequential boundary without proof. Credentials, data destinations, deployment authority and irreversible actions come before tidying documentation. Next, repair missing acceptance and recovery controls. A clean pattern library matters, but it should not outrank a release path that cannot stop an untested change.
Turn repeated judgment into the narrowest reliable mechanism. That may be a linter rule, dependency policy, test, permission boundary, approval check or runbook requirement. Do not install a giant platform because one repository lacks one refusal. Start with the control that closes the demonstrated path.
Assign an owner and an expiry condition to every exception. Some work will need to proceed while a repair is underway. Record who accepted the risk, what evidence they saw and when the exception stops applying. Otherwise a temporary bypass becomes the repository's most dependable route.

When is the repository audit finished?

The audit is finished when the agreed scope has evidence for every result and the repaired controls survive a rerun. Closing a ticket is not the finish line. Run the same failing case that exposed the gap, then run the allowed case and preserve both outcomes beside the finding.
Keep the audit executable where possible. Version its checks with the repository, run them on relevant changes and make failures point to the rule and the repair path. When production reveals a new failure, add a reproduction case so the next audit checks what the factory has actually learned. That is how quality becomes a closed loop instead of a quarterly document exercise.
Start with one repository and one real route to production. Map it, challenge its controls and fix the most consequential refusal that is missing. Then rerun the evidence. The goal is not a badge. It is a factory that can say no before bad work becomes somebody else's incident.

Quick answers

Is an agent instruction file enough for governance?

No. It can explain intent and local rules, but mechanically detectable violations need controls that can stop the work.

Should every repository use the same audit checklist?

No. Reuse control families, then set requirements from the repository's data, authority, release path and failure consequences.

How often should the audit run?

Run executable checks on relevant changes and repeat the broader route audit when permissions, release machinery or operating risks change.