A Red Build Needs a Diagnosis

Preserve the failing command, code revision, check configuration and diagnostic output before retrying or editing anything. Continuous integration, or CI, runs automated checks as changes come together. A regression is behavior a change breaks. The baseline is the unchanged revision used for comparison. A red status alone doesn't tell you which of those pieces failed.

In her account of experimenting with agentic development, Rachel Laycock describes an application that ran while its build repeatedly failed. She leaves the causes open, including possible interference between agents and pressure on the checking pipeline. That's a reason to investigate the delivery machinery, rather than dismiss experienced developers' concern.

Gunner Technology recommends making diagnosis an explicit job in the factory. Attach the dependency versions, test setup and failure details to the affected change. Keep credentials and private records out of routine logs. The next agent should receive a reproducible problem, with permission to investigate, rather than an instruction to make the status green somehow.

Run the unchanged revision and the proposed change in separate clean environments using comparable inputs and the approved checks. Freeze the baseline revision; a moving branch name isn't a reproducible reference. Record the runtime, dependencies, service versions, configuration and test data for each run. Explain any differences the proposed change requires.

Use isolated test data and services where the work requires them. If the runs share a database, an agent testing its patch can alter the conditions the baseline receives. Preserve the relevant test order and concurrency settings too. A comparison between a quiet local run and a crowded CI worker hasn't isolated the code change.

When a new test or changed dependency cannot run against the old revision, record that limitation and design a compatible reproduction. Don't silently alter the requirement to get matching results. Our post on giving agents a real system to test explains the environment boundary; here that boundary has to hold across the comparison.

Failure on both revisions establishes that the proposed change isn't required for that observed failure. It doesn't establish that the causes match or that the patch introduced no other defect. Compare the actual errors, execution paths and failed expectations. A shared job name or exit status is too little evidence to settle responsibility.

Consider a hypothetical export change tested while its database fixture cannot start. Both revisions fail during setup. The new patch could still drop a required field from the export; neither run reached the behavior that would reveal it. Repair the fixture, then run the relevant checks against both revisions again. Calling the first failure pre-existing hasn't tested the export.

If the baseline passes and the candidate fails under matched conditions, investigate the changed code and its interactions. If neither result is stable, keep the cause unresolved. Separate a suspected product regression from an unavailable environment or an unreliable check, and route each problem to the component that needs repair. Use the diagnosis to assign repairs. Release still requires valid evidence and approval.

Retries help when they test a recorded hypothesis under a bounded investigation plan and retain the earlier failures. A flaky test sometimes passes and sometimes fails without an intentional change. The pytest documentation identifies uncontrolled state, test ordering and parallel execution as potential causes. An intermittent result deserves an explanation, not an automatic exemption.

Repeat the same revision and setup before changing variables. Then investigate a suspected shared resource, test order or timing dependency while recording what changed. Have an independent verifier repeat the checks after a repair. A passing retry can help reveal a pattern, but it cannot explain away the failed attempt that came before it.

Keep the builder from removing assertions, disabling required checks or raising a threshold merely to finish its ticket. A suspected test defect can become a separately reviewed repair. Any permitted temporary exception needs its own authorized decision. As Missing Evidence Cannot Pass a Release explains, the absence of usable proof must stay visible.

Hold changes that depend on the broken check while the factory diagnoses and repairs it. Independent work can continue within its existing requirements. If a shared environment keeps failing, reduce admissions to the affected route instead of generating more patches that cannot be judged. The queue needs a working inspection station.

Start with a recurring failure in a repository you own. Give the factory a clean baseline, isolated comparison runs and a place to preserve every result. Require the repaired checking route to demonstrate that it accepts valid work and rejects a known defect. Then recheck the waiting changes before presenting them for approval.

People decide what correct means and whether the returned work is right. Agents can reproduce failures, trace causes and carry out approved repairs. The factory should make that investigation repeatable. A permanently red build consumes everyone's attention while settling nothing; restoring its meaning is engineering work worth doing.