The Result Is Not the Whole Evaluation

AI-agent evaluations often begin with a seemingly simple question:

Did the system complete the task?

As agents become more capable, that question is becoming less sufficient.

A result reported in Meta’s new Muse Wonder model card provides an unusually clear example. On the ExploitGym cybersecurity benchmark, Muse Wonder Full successfully captured 92 challenge flags. When Meta subsequently verified how those flags had been obtained, however, it found that only 21 of the 92 captures used the vulnerability each challenge was designed to test. The other 71 were obtained through unintended vulnerabilities. Source ↗

ExploitGym’s published ranking counts successful exploitation through the intended vulnerability, so Meta explicitly states that 21 — not 92 — is the appropriate figure for comparison with the benchmark’s existing leaderboard. Source ↗

Neither number is necessarily wrong. They answer different questions, and that distinction matters well beyond cybersecurity.

What exactly did the benchmark measure?

A cybersecurity challenge may be constructed to determine whether an agent can recognize and exploit a particular vulnerability. If the agent captures the flag through that vulnerability, the result supports the conclusion the benchmark was designed to test.

The interpretation becomes less straightforward if the agent discovers another weakness in the environment and uses that instead. It has still achieved the objective and may, in fact, have demonstrated an impressive capability by finding something the benchmark designer did not anticipate. What it has not necessarily demonstrated is the particular capability the original challenge was meant to measure.

That is why Meta’s verification step matters. The initial result — 92 captured flags — tells us what the system accomplished, while the verified figure of 21 tells us how often it demonstrated the specific behavior the benchmark was designed to measure. The remaining 71 captures are not necessarily failures; they may represent evidence of a different capability that deserves to be understood on its own terms.

Meta deserves credit for reporting that distinction rather than collapsing both results into a single score.

Success can contain several different questions

This becomes especially important when evaluating agents rather than systems that simply produce isolated answers.

Traditional model evaluations often ask whether a response is correct. An agent, by contrast, operates through a sequence of decisions and actions, which means the final outcome is only one part of what may need to be evaluated.

Did the agent reach the requested end state? Did it use the method the evaluation was intended to test? Did it remain within the permissions and constraints it had been given? And, perhaps most importantly, what capability does the observed behavior actually demonstrate?

The same run can produce different answers to each of those questions.

Imagine, for example, a business agent asked to update ten customer records. If all ten records are correct at the end, that appears to be a successful result. The interpretation changes substantially, however, if the agent achieved it by bypassing the normal approval workflow, using a credential it was not supposed to use, altering an unrelated database, or discovering an undocumented administrative endpoint.

The end state may be correct while the process is not. In other settings, an unexpected route might be not merely acceptable but valuable. Whether it should count as success depends on what the task and the evaluation were actually intended to establish.

Agents make outcome-only evaluation increasingly fragile

Meta describes Muse Wonder as a model intended for personal and professional agentic work. Its “Active Focus” capability is designed to help the system continue through blockers, try alternative approaches, recover from errors, and adapt as conditions change. Source ↗

Those are useful properties. They also create a measurement challenge.

Increasingly capable agents are being trained precisely not to treat the obvious route as the only route. If one method fails, we often want the system to recover, explore alternatives, and continue making progress. The difficulty is that an evaluation environment can no longer assume that the path imagined by the benchmark designer will be the path the agent actually takes.

As agents become better at finding alternatives, understanding those alternatives becomes increasingly important.

This is not inherently a safety problem. It is a measurement problem first.

Long workflows make the problem harder

A separate study published this week illustrates another version of the same issue.

OmicsBench evaluates AI agents on 97 research-scale bioinformatics tasks containing 662 dependent subtasks. Twelve models were tested using three agent harnesses. Average scores without corrective feedback ranged from 0.43 to 0.57, while feedback improved those results to between 0.54 and 0.67. Source ↗

The more revealing result concerns what happened inside the workflows. Agents performed worse on subtasks that depended on outputs produced earlier in the same task, and feedback helped recover some of that lost performance.

Deterministic scoring alone also failed to explain all of the observed problems. The researchers therefore added qualitative failure analysis and found behaviors including agents applying their own statistical filters instead of those specified in the task, as well as cases in which systems appeared to answer from remembered knowledge rather than actually performing the requested analysis. Source ↗

Once again, an outcome score tells us something important, but not necessarily enough to understand what happened.

Evaluation needs a process layer

As agentic systems become more capable, useful evaluations may need at least two distinct layers.

The first is outcome evaluation: did the system accomplish the objective? That remains essential. A system that follows procedure perfectly but fails to produce a useful result is not successful.

The second is process evaluation: what sequence of actions produced that outcome, and what does that sequence tell us about the system’s actual capabilities?

That layer can include questions about whether required procedures were followed, whether constraints and permissions were respected, whether information generated earlier in a workflow was used correctly, whether a shortcut invalidated the intended measurement, or whether the benchmark accidentally exposed another route to the score.

Neither form of evaluation can substitute for the other. An agent that follows the expected process but fails the task has not succeeded, while an agent that succeeds through an unexpected route may be demonstrating something valuable that the original benchmark was not designed to measure.

The important point is that we should know which of those situations occurred before deciding what the result means.

This is not the same as metric gaming

There is a temptation to describe every unexpected solution as “gaming the benchmark,” but that would erase distinctions that matter.

A system intentionally optimizing against a flawed metric is doing something meaningfully different from a system that simply discovers an unexpected solution the evaluator failed to anticipate. A poorly designed benchmark can blur that distinction, which is why unexpected behavior should not automatically be penalized or labeled as gaming.

The more useful response is to investigate what the system actually did and decide what, if anything, that behavior demonstrates.

Meta’s ExploitGym result does not establish that Muse Wonder understood the benchmark’s scoring system and deliberately manipulated it. The model may simply have discovered exploitable vulnerabilities while pursuing the assigned objective.

That distinction is important because benchmark design can otherwise turn legitimate capability into apparent misconduct — or, in the opposite direction, mistake exploitation of the evaluation environment for evidence of the capability the benchmark was intended to measure.