Comparing Agents

Run an agent across the scenarios and personalities you care about and get a pass-rate verdict for each slice of your coverage. Then lay two reports side by side to see whether another agent, or a new version of the same one, does better on the same tests.

A comparison report answers two kinds of questions:

• Is this agent ready to ship? A single report gives a pass-rate verdict per slice of your coverage.
• Which agent does better? Two reports laid side by side show exactly which tests improved or regressed - whether you compare two different agents or two versions of the same one. See Comparing two reports further down.

Creating a comparison

You organize a comparison into groups. Each group picks a set of scenarios and one or more personalities, and every scenario in the group runs against every personality in it. Groups let you label different slices of your coverage (for example “New customers” vs “Existing customers”) and get a verdict for each.

1. Go to the Compare page in the left sidebar and click New comparison.
2. Pick the agent being evaluated. To compare two agents, create one report per agent with the same groups, then lay the reports side by side afterwards.
3. Add one or more groups. For each group, give it an optional name and pick the scenarios and personalities it covers.
4. Optionally set Repeat each test (1-10) to surface flaky behavior.
5. Click Create comparison.

Voxli runs every scenario in each group against every personality in that group and generates one test result per pair.

Reading the results

The results page opens with an insights row at the top showing the average score, how many tests have passed, and the number of blocker failures and warnings. Below that, each group is rendered as its own matrix of scenarios and personalities.

• Green pass rates (90% or above) indicate the group is ready to ship.
• Yellow (70-89%) indicates review-worthy issues.
• Red (below 70%, or any blocker failure) indicates a critical problem in that group.

Group names can be edited inline by clicking the title. Click a scenario row to expand it into its individual tests. Click any cell to open the full test result with the conversation transcript and assertion outcomes.

Choosing which metrics to show

Click Settings (the sliders button) above the matrix to control what each row reports. Under Metrics, pick one or more measurements - each one you select becomes its own column:

• Score - the weighted pass rate per test. Always available.
• Hallucinations - the number of unsupported or contradicted claims, when hallucination detection is on.
• Custom metrics - anything you have defined under Settings > Metrics, such as response time, token usage, or cost. Each metric and each of its aggregates (for example, an average and a p95) is offered separately.

The selector lists every metric you have configured, so you can read the comparison along one dimension at a time. To add more measurements, define them first - see Custom Metrics. The same Settings menu also switches whether personalities are laid out as columns or rows.

Your choices - the selected metrics, expanded scenarios, and layout - are stored in the page URL. That makes a configured view shareable, and it is restored when you come back to the report.

Repetitions and flaky tests

Setting Repeat each test above 1 runs every test that many times. Repetitions surface flaky behavior: an agent that passes a test four times out of five has a problem a single run would not reveal.

In the matrix, a repeated cell shows one headline number. For scores, this is the average across its repetitions. For hallucination counts, it’s the total across all repetitions. Click the cell to see the individual repetitions - a breakdown opens that lists each repetition with its score, whether it passed or hit a blocker, and a link to its full result.

A test whose repetitions disagree - some pass, some fail - is flaky. The matrix still shows a single headline number for it (average score or total hallucinations), so use the breakdown to spot the disagreement. In the insights row at the top, a test only counts as passed when every one of its repetitions passed.

To audit every repetition at once, click Settings and turn on Show repetitions. Every test then expands into one row per repetition, each linking to its result. The option appears once a test has run more than once.

Comparing two reports

Once you have more than one finished comparison, you can lay two of them side by side to answer “did readiness improve since last time?” or “how does agent X stack up against agent Y on the same scenarios?”.

1. On a report, click Compare to (next to Re-run).
2. Pick another completed report. The picker only lists reports with the same configuration - the same scenarios and personalities - since the agent is the thing being compared. Each option leads with the report name, with its agent and run time beneath. Search by agent or report name to narrow the list.

The view becomes a side-by-side diff. The report you picked is the reference (A) and the current report is the subject (B). Each metric becomes a three-column group: the A value, the B value, and a colored B - A delta - green where B improved, red where it got worse, and neutral for a change within the metric’s margin. Personalities are shown as rows in this mode. Anything that exists in only one report is kept and badged New or Removed.

Clear the comparison with the ✕ next to the picker to return to the single-report view.

Re-running comparisons

After making changes to an agent (or the underlying scenarios) you can re-run an existing comparison to get fresh results. Re-run opens the New comparison dialog pre-filled with the same configuration, so you can adjust it if needed and track progress over time.

What’s next

• How Metrics Are Summarized - how the summary and average numbers are calculated.
• Custom Metrics - define the measurements the metric selector offers.
• Personalities - shape how the simulated user talks across groups or configurations.