Running Tests

This guide shows how your code drives each test’s conversation through Voxli’s REST API. Runs start in Voxli, and a GitHub or Local agent hands your code the IDs of the test results to drive.

Driving a Test Result

For each test result:

• Start the conversation by calling POST /test-results/{id}/next-message to get the first tester turn
• Enter a conversation loop where you relay each turn between Voxli and your agent, recording your agent’s replies with POST /test-results/{id}/conversation
• Continue until Voxli signals end_chat: true

Each next-message response may return ready: false if the next turn is not yet available. Poll the endpoint with a short delay until it returns ready: true.

Each ready turn contains either message (free text from the tester) or action (an invocation of one of the actions your chatbot registered for this turn). Branch on which one is populated - they are mutually exclusive. See Tools, Events, and Actions for how to register actions.

Each run is automatically marked as completed once all its tests finish. GET /runs/{id}/results lists a run’s test results with their status and score.

Message Metadata

When posting agent messages to the conversation endpoint, you can include a metadata object with performance metrics. Voxli reads the following recognized keys:

import time

start = time.monotonic()
agent_response = get_agent_response(tester_message)  # your agent logic
response_time_ms = round((time.monotonic() - start) * 1000)

requests.post(conversation_endpoint, headers=headers, json={
    "type": "message",
    "content": agent_response,
    "metadata": {
        "responseTime": response_time_ms,
        "inputTokens": input_tokens,
        "outputTokens": output_tokens,
        "cost": cost,
    }
})

When available, these metrics are displayed as averages in test result details and comparison views. They help identify performance regressions and cost differences across agent configurations.

Attaching Data After the Chat

By default, Voxli starts processing and scoring a result as soon as the conversation ends. If your metrics or tracing data arrive after the chat is over, switch the test result to manual completion with PATCH /test-results/{id} and body {"completionMode": "manual"}. The change must land before the conversation ends. The conversation then ends as usual (end_chat: true), but the result stays in running and waits for you. Attach the remaining data, then finish the result yourself:

"""
Attach data after a chat ends, then finish the result yourself.

1. Switch a test result to completionMode "manual"
2. Run the conversation loop until end_chat
3. Attach post-chat data to the conversation
4. Finish the result to start scoring
"""

import os
import time
import requests

api_key = os.getenv("VOXLI_API_KEY")
base_url = os.getenv("VOXLI_API_URL", "https://api.voxli.io")
result_id = os.getenv("VOXLI_TEST_RESULT_ID")  # from a run you started

headers = {"Authorization": f"Bearer {api_key}"}

# 1. Make the test result wait for an explicit finish call. This must land
# before its conversation ends.
requests.patch(f"{base_url}/test-results/{result_id}", headers=headers, json={
    "completionMode": "manual",
})
conversation_endpoint = f"{base_url}/test-results/{result_id}/conversation"

# 2. Run the conversation loop as usual (see the complete example).
# The endpoint returns the recorded entry, with its server-generated id.
# Keep the ids you care about; they are also returned by GET /test-results/{id}.
reply = requests.post(conversation_endpoint, headers=headers, json={
    "type": "message",
    "content": agent_response,
}).json()
reply_id = reply["id"]

# ... keep relaying turns until next-message returns end_chat: true ...

# 3a. Merge late data into an earlier message, for example metrics or
# tracing details that your observability stack delivers after the fact.
requests.patch(f"{conversation_endpoint}/{reply_id}", headers=headers, json={
    "metadata": {
        "responseTime": 840,
        "traceId": "trace-8c14",
    }
})

# 3b. Insert a tool call at the position where it actually happened.
requests.post(conversation_endpoint, headers=headers, json={
    "type": "tool",
    "name": "lookup_order",
    "metadata": {"order_id": "A-1042"},
    "after": reply_id,
})

# 3c. Internal events recorded now are visible to scoring but were never
# part of the chat, so they cannot influence the conversation itself.
requests.post(conversation_endpoint, headers=headers, json={
    "type": "internal-event",
    "name": "session_summary",
    "metadata": {"handoff": False, "sentiment": "positive"},
})

# 4. Finish the result. Scoring runs in the background, so poll the
# result until its status turns "completed".
requests.post(f"{base_url}/test-results/{result_id}/finish", headers=headers)

while True:
    status = requests.get(
        f"{base_url}/test-results/{result_id}", headers=headers
    ).json()["status"]
    if status in ("completed", "failed", "canceled"):
        break
    time.sleep(2)

print(f"Test result {result_id} finished with status {status}.")

Message IDs

Every conversation entry carries a server-generated id. You get it back from the conversation endpoint’s response, from GET /test-results/{id}, and, for tester turns, as message_id in the next-message response. Use these ids to address messages in the two operations below. If you need your own correlation keys, put them in the message metadata.

Updating a Message

PATCH /test-results/{id}/conversation/{message_id} merges a metadata object into an existing message. Top-level keys overwrite existing ones; other keys are preserved. The merged metadata must stay under 100KB, and the actions key cannot be changed after recording.

Inserting a Message

The conversation endpoint accepts an optional after or before field with a message id. The new entry is inserted at that position instead of appended, so a tool call or event that surfaced late can sit where it actually happened. The two fields are mutually exclusive, and inserted messages cannot register actions.

Finishing the Result

POST /test-results/{id}/finish starts processing and scoring in the background and returns immediately with status 202, the result’s id and its status. Poll GET /test-results/{id} until the status turns completed. The call is rejected while the conversation is still going (400) and once the result is already completed, failed, or canceled (400).

A few rules to keep in mind:

• Finish all conversation writes before calling finish. Updating and inserting messages is rejected (409) from that call onwards, because scoring reads the conversation as recorded.
• Finishing a result that is already being processed is rejected (409), so a retried finish never scores the same conversation twice.
• Nothing times out. A manual result waits until you finish it, and its run stays running until then. If you abandon a result, cancel it with PATCH /test-results/{id} and body {"status": "canceled"}.

Retrying a Test Result

If a finished test result is unusable — for example, the conversation ended on a fallback handler or hit a transient error — you can retry it in place. A single call clones the result into a new pending attempt on the same run, cancels the original (so it drops out of the run’s score average), re-opens a completed run, and dispatches execution server-side:

new_result = requests.post(
    f"{base_url}/test-results/{result_id}/retry",
    headers=headers,
).json()
# 201 Created: new_result is the new attempt's id and status (pending), and
# the Location header is its URL. It is attached to the same run as the
# original, which is now canceled. Its metadata.previous_test_result_id
# points back to the original.

The clone re-runs the identical generated case and inherits the original’s test, run, agent, personality, instruction, and assertion criteria. Retrying is supported for all agent types. The result must be finished (completed, failed, or canceled); retrying a result whose run was manually canceled is rejected.

Execution is dispatched for you per agent type: hosted agents run server-side, GitHub agents trigger their workflow, and Local agents pick up the new attempt the next time their runner connects.