AI coding agents can produce changes, run tests, and report success with remarkable speed. Yet a fundamental question remains when the work is handed back to a person: Did the agent actually accomplish the requested task?
A passing test suite can answer part of that question. So can a clean Git diff, a recorded execution trace, or evidence that an agent stayed within approved file boundaries. None, on its own, demonstrates that the user's intended outcome was achieved.
This distinction became central to Velorin Intelligence's work on Constraint Sentinel, an experimental verification tool whose product development we paused in November 2026. Our reason was not that verification had proved impossible or useless. It was that we had not established sufficient incremental practical value over simpler checks to justify continuing the product.
What Sentinel could establish
Constraint Sentinel began with a narrow, defensible proposition: compare a coding agent's resulting Git changes against file boundaries approved at a trusted baseline. If a task permitted edits to cart calculations but protected payment-processing files, the verifier could inspect whether those boundaries were respected.
That is a meaningful property, but it is not task correctness. An agent can stay entirely inside the permitted files and still calculate the wrong total. A scope check cannot infer the business requirement that was omitted from the task or the test suite.
During subsequent research, we examined additional verification mechanisms, including whether newly introduced tests were actually collected and executed, whether existing tests disappeared, and whether test bodies contained a trivially vacuous assertion. These mechanisms are useful in particular circumstances. They also raise a harder question: what does the extra checking add beyond ordinary continuous integration and a small set of owner-defined requirements?
The experimental layered checker discussed below was a research combination built around an unchanged Sentinel probe, not the published Constraint Sentinel beta. We do not attribute its full capabilities to the historical product release.
Three experiments, one persistent boundary
In AV-01 we compared ordinary pytest execution, a minimal owner-defined contract, and a layered research verifier across 18 deliberately constructed code changes. Ten were labeled invalid and eight legitimate. The minimal contract flagged three of ten invalid cases; the layered checker flagged eight. But it also raised alerts on four of eight legitimate changes that the simpler check left alone.
Those figures describe an intentionally selected synthetic test set, not real-world accuracy. Most of the layered checker's extra detections concerned process obligations: an uncollected test, an unauthorized test removal, a skip, a vacuous assertion, or a change that silently removed a test from collection. These are important findings, but they are not the same as detecting incorrect program behavior.
AV-02 added narrowly scoped, externally supplied owner exceptions. On the selected cases, four explicit grants eliminated the four legitimate-case alerts without suppressing the eight invalid-case alerts. That was a useful mechanism demonstration, not evidence that the overall system had become less burdensome. An owner still had to understand and authorize the exceptions. One permissive example also illustrated that silencing an alert can require relaxing the underlying quality obligation.
AV-03 regenerated 18 fresh synthetic changes around the same known failure mechanisms and reproduced the aggregate pattern: eight of ten invalid cases flagged by the layered checker, with four of eight legitimate changes flagged until owner exceptions were supplied. This is a nonindependent reproducibility check, not a blinded, independently authored replication. The planned human-review stage remains incomplete: no genuine reviewer-time or owner-authorization-time observations have been collected.
Across these experiments, two cases remained especially instructive. One change introduced a functional regression that passed the available tests. Another supplied a test bearing the expected name but checking the wrong boundary. Every verification configuration missed both. The evidence showed that selected checks ran; it did not establish that the requirements those checks failed to express had been satisfied.
The wider research reaches a similar concern
These findings sit within an increasingly substantial body of research on the difference between agent activity, correct outcomes, and the work left to humans.
A 2026 observational study by Tang and colleagues analyzed 20,574 real coding-agent sessions. Among the misalignment episodes with visible resolutions, 91.49% required explicit user correction. That is not a 91.49% failure rate across all sessions; it describes how observed problems were resolved. The result nevertheless makes clear why apparent agent autonomy can conceal significant human intervention.
Dhanorkar and colleagues, interviewing 17 experienced developers, characterized oversight as more than reviewing final output. It also includes advance controls, joint planning, real-time monitoring, and retrospective review. Our owner-exception experiments touched precisely this trade-off: a warning can be avoided by specifying intent earlier, but specifying that intent has a human cost.
Productivity evidence is mixed, which matters. METR's randomized study of experienced open-source developers found that early-2025 AI tools increased completion time by 19% in its particular setting. In February 2026, METR reported signs that later tools may be more helpful, while emphasizing that selection effects made the newer estimates unreliable. Meanwhile, a 2026 NBER working paper studying more than 500,000 developers reported large associations between agent adoption and commit activity but substantially smaller effects at the release stage. These studies measure different populations and outcomes; they should not be collapsed into a claim that AI coding tools generally slow developers down.
The adequacy of the tests themselves is also an established concern. OpenAI's July 2026 audit estimated that roughly 30% of SWE-bench Pro tasks had consequential evaluation problems, including underspecified requests and low-coverage tests. A benchmark pass is only as meaningful as the specification and evaluator behind it.
There are promising results on how to use human attention more selectively. A CHI 2026 study of when users should confirm multistep agent actions found that intermediate confirmation scheduling reduced average completion time by 13.54% relative to confirmation only at the end, in its controlled setting. The lesson is not that all supervision is waste. It is that where, when, and why the human intervenes can determine whether supervision helps.
The missing economic result
A verifier has a cost even when its software is free. It may require rules to be authored, decisions to be approved, alerts to be adjudicated, false positives to be investigated, and policies to be maintained. It also consumes runtime and creates another component that must itself be trusted.
For the incremental verifier to be worth adopting, the practical value of additional consequential failures caught must outweigh those costs. We have not measured that inequality for Constraint Sentinel. AV-03's automated runs cannot substitute for real human review-time observations, and neither the observed synthetic detection counts nor local runtime figures establish commercial return on investment.
This is why we paused development. The historical beta remains available as experimental open-source material, while the broader research prototypes and negative findings remain valuable as evidence. A working mechanism is not, by itself, a viable product.
What a better evaluation would count
Future AI workflow evaluations should be organized around independently established task outcomes and total human effort, not only passing checks or agent-generated activity. Useful measures include time to a demonstrably correct result, number and duration of human interventions, repeated clarification, rework after an apparent completion, severe failures prevented, and the effort required to configure and maintain oversight.
This does not mean discarding Git controls, tests, or execution evidence. These are useful components of responsible engineering. It means keeping their claims narrow: a test ran; a file was not changed; a declared requirement was checked. If correctness has not been independently established, the status should say so.
Constraint Sentinel's most useful result may therefore be the decision not to extrapolate from successful mechanisms to unsupported assurance or product value. The research did not resolve the economics of oversight. It clarified the evidence we would need before making that claim.
Verification is evidence about a process. Completion is a claim about an outcome. Responsible agent systems must not confuse the two.
Sources and disclosure
Velorin primary record. Constraint Sentinel research repository (historical experimental beta; product development and beta recruitment paused November 2026). AV-01, AV-02 and AV-03 reports informed the analysis. AV-03's human-review stage remains unrun; its master archive includes evaluator-only materials not intended for public release. The layered research checker should not be confused with the published Sentinel beta.
Selected external research
- Tang et al. (2026), How Coding Agents Fail Their Users, EMNLP 2026 — observational coding-agent misalignment study.
- Dhanorkar, Passi & Vorvoreanu (2026), Human oversight of agentic systems in practice — interviews with 17 experienced developers; preprint.
- Becker et al. (2025), METR randomized productivity experiment and February 2026 update.
- Demirer, Musolff & Yang (2026), Writing Code vs. Shipping Code, NBER Working Paper 35275, September 2026 revision — observational matched event-study estimates.
- OpenAI (July 2026), Separating signal from noise in coding evaluations — SWE-bench Pro task-quality audit.
- Zhou et al. (2026), When Should Users Check?, CHI 2026 — controlled confirmation scheduling study.
- Kapoor et al. (2024), AI Agents That Matter — accuracy, cost and evaluation quality.
Method and claim boundary. The AV experiments used small, researcher-designed, nonrepresentative synthetic changes and did not involve external coding-agent output or a production deployment. AV-03 reproduced known failure mechanisms in fresh fixtures but was not independently blinded. A successful local verification mechanism is not proof of a product's effectiveness, security or return on investment. The artwork is illustrative, not experimental data.