It Passed Once. Would It Pass Five Times in a Row?

It Passed Once. Would It Pass Five Times in a Row?

What a 2026 voice-agent benchmark says about reliability, policy adherence and the gap between a good demo and a dependable call

A 2026 benchmark of 12 voice agent systems found that 72.2% of conversations with a completed task still contained at least one policy or grounding deviation, and that results fell sharply when the same scenario was repeated. What it means for testing voice agents.

In brief

1. A May 2026 preprint from ServiceNow researchers, EVA-Bench, tested 12 voice agent systems on 213 scenarios, running each five times.
2. Two results stand out. Peak performance and reliable performance were far apart (a median gap of 0.44 on accuracy), and 72.2% of conversations where the task was completed still contained at least one faithfulness deviation.
3. The benchmark uses simulated callers and mock tools, so it is not a ranking of products or a prediction of live performance.
4. For teams deploying voice agents, the lesson is structural: test every scenario repeatedly, and score policy adherence separately from task completion.

1. What the study did

The paper, “EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents,” is by Tara Bogavelli and colleagues at ServiceNow. It is an arXiv preprint (arXiv:2605.13841, revised May 27, 2026) and we found no peer-reviewed venue named. ServiceNow sells enterprise software, so read it as vendor-authored research, though the authors released the framework, scenarios and data as open source.

The setup:

1. 213 scenarios across three enterprise domains: airline customer service (50), healthcare human resources service delivery (83), and IT service management (80).
2. 12 systems of three types: seven cascade systems (separate speech recognition, language model and speech synthesis), two hybrid systems, and three speech-to-speech systems.
3. Bot-to-bot audio conversations. A simulated caller speaks to the agent, and conversations where the simulator misbehaved are regenerated before scoring. 12.0% of trials across four systems had to be regenerated.
4. Five trials per scenario on clean audio, and three trials on a 90-scenario subset for noise and accent tests.

It scores two things. EVA-A measures accuracy: whether the task was completed (checked against the final database state), whether the agent stayed faithful to policies and tool results, and whether its spoken output was accurate, including entities such as confirmation codes and dollar amounts. EVA-X measures experience: whether the conversation progressed, whether responses were concise enough for speech, and turn-taking.

2. Result one: working once is not working every time

The paper reports three ways of counting success on the same trials:

1. pass@1: the average share of trials that pass.
2. pass@k: the share of scenarios where at least one of the k trials passes. This is peak performance, what a demo shows.
3. pass^k: the probability of passing all k trials. This is reliability, what a customer base experiences.

The median gap between pass@k and pass^k was 0.44 on accuracy and 0.24 on experience. Across the twelve systems, the accuracy gap ranged from about 0.26 to about 0.65 (from the paper’s table as extracted).

An illustration, ours and invented, not from the paper: if an agent handles a given scenario correctly 80% of the time, the chance it handles that scenario correctly five times in a row is about 33% (0.8 to the fifth power). A team that tests each scenario once and sees a pass would reasonably call it done. A customer base that calls repeatedly meets the other 20%.

Also from the paper: no system exceeded 0.5 on both accuracy and experience at pass@1. Only one cleared 0.4 on both. These scores come from this benchmark’s scenarios and scoring, so they should not be read as absolute quality ratings.

3. Result two: the task got done, but the rules were not followed

The faithfulness metric checks whether the agent’s actions stay grounded in its instructions, its policies, the tool results and what the caller said. It is judged by a language model, not a human.

The paper found that 72.2% of conversations where the task was completed still contained at least one faithfulness deviation. The authors cite this as evidence that task completion does not predict faithfulness.

What that means in plain terms: a database check can say “the booking exists” while the conversation involved an agent that skipped a required check, stated a policy wrongly or promised something it should not have. A pass on outcome can hide a failure on conduct. We have not seen the individual deviations and do not know how severe they were, since the metric flags any deviation below perfect.

4. Other findings worth knowing

1. Speech accuracy tracks success. Among cascade systems, the accuracy of transcribing key entities correlated strongly with task completion (r = 0.93, p = 0.002, across 7 systems). Systems below 70% key-entity accuracy averaged 0.37 task completion, versus 0.60 for those above. With only seven systems, treat this as a pattern, not a law.
2. Accents and noise hurt cascade systems. French-accented speech lowered cascade task completion by about 10 points on average (worst system, 17). Combined accent and noise lowered it by about 19 points (worst, 31). Speech-to-speech systems showed no significant accuracy degradation under the accent test.
3. Turn-taking is fragile. It was the metric most sensitive to perturbation, with 81% of measurements showing significant degradation.

5. What the authors say limits the findings

These are the authors’ own caveats, and they matter:

1. Simulated callers are assumed to stand in for real ones, which may not hold. The simulator does not generate interruptions.
2. Tools are mocks, so real API failures and latency are not captured.
3. Scenarios are English only. The accent test used one accent and one noise environment.
4. Faithfulness is judged by language models, which can show bias, especially when judge and system share a model family.
5. Task completion is binary, with no partial credit.
6. With 12 systems, broad claims about model classes are not supported, and prompts were not optimized.

6. What to do about it (ours)

This is our reasoning, not a finding from the paper. It follows one principle: test the flow, not the single conversation you happened to see.

1. Run every scenario several times. One pass proves little. Report both the average and the all-trials rate, and watch the gap.
2. Score conduct separately from outcome. Check whether required verification steps happened, whether stated policies were correct, and whether anything was promised that the system cannot deliver.
3. Test varied phrasing, not a fixed script. Callers say the same thing in dozens of ways. A fix that makes one transcript pass is a patch, not a repair.
4. Include accents, noise and interruptions. The benchmark’s own limits suggest these are under-tested everywhere.
5. Check spoken details separately. Names, codes, amounts and dates deserve their own measure.
6. Validate your simulator. If you use simulated callers, check they behave like yours before trusting the scores.
7. Use real calls as the final check. Sample live calls against the same criteria.

7. A scorecard for your own testing

What we do not know

1. How any particular commercial voice agent performs. The study evaluates configurations of models, not deployed products, and we have not named any.
2. How results change on real telephone audio, with real systems behind the agent, or in languages other than English.
3. How severe the faithfulness deviations were, and how often a human reviewer would agree with the language-model judge.
4. Whether the findings hold after peer review.

This article is general information, not advice. We read the preprint and did not rerun its tests.

See it working

Not sure how your AI conversations are really performing? GoZupees works with teams to track quality, handoff and outcomes. If that is on your roadmap, book a demo at www.gozupees.com.