Local

Run Voxli tests against your bot through a bridge on your computer.

The CLI registers your machine as a local agent, receives assigned tests, and runs your bridge command. Setup is complete after one test finishes scoring and you inspect its conversation.

Write your bridge

Identify your bot’s entry point, conversation state, tool execution hooks, and required settings. Save the sample as run_test.py and install requests and python-dotenv with your project’s package manager. For a standalone Python project:

python -m pip install requests python-dotenv

Adapt get_agent_response(session_id, turn) to call your bot. Keep state across turns and isolate it per test using session_id. For text, pass turn.get("message") or "". If turn["action"] is present, apply its name and arguments in your bot. A text-only adapter should raise an error for unsupported actions. Give the bot’s own network calls timeouts too.

Return an ordered list of conversation entries. Each reply is a {"type": "message", "content": "..."} entry. Keep multiple replies and events in their original order. Tool entries belong before replies that use their output. See Tools, Events, and Actions for tool recording and action registration. Report any limits on access to the bot’s tools or traces.

The CLI passes these environment variables to your command:

VOXLI_API_TOKEN: authentication injected automatically after voxli auth
TEST_RESULT_IDS: JSON array of assigned test result IDs

The GitHub integration runs this same script. Its workflow maps the test_result_ids input to TEST_RESULT_IDS and supplies a Voxli API key.

"""Relay Voxli conversations to your bot locally or in GitHub Actions.

Requires requests and python-dotenv. Adapt get_agent_response before running.
VOXLI_API_TOKEN and TEST_RESULT_IDS come from the CLI or workflow environment.
"""

import json
import os
from pathlib import Path
import time

from dotenv import load_dotenv
import requests

SESSION_DEADLINE_SECONDS = 90 * 60
MAX_TURNS = 200
POLL_INTERVAL_SECONDS = 2
HTTP_TIMEOUT_SECONDS = 60


def get_agent_response(session_id: str, turn: dict) -> list[dict]:
    """Call your bot and return ordered Voxli conversation entries.

    Keep state per session_id. Handle turn["action"] when present, otherwise
    send turn.get("message") or "". Put tool entries before replies using them.
    """
    raise NotImplementedError("Connect your bot and its tool execution traces here")


def load_bot_settings() -> None:
    path = Path(os.environ.get("BOT_ENV", Path(__file__).with_name(".env")))
    if "BOT_ENV" in os.environ and not path.is_file():
        raise FileNotFoundError(f"Bot settings file does not exist: {path}")
    load_dotenv(path, override=False)


def voxli_request(
    client: requests.Session,
    method: str,
    url: str,
    body: dict | None = None,
    timeout: float = HTTP_TIMEOUT_SECONDS,
) -> requests.Response:
    response = client.request(method, url, json=body, timeout=timeout)
    response.raise_for_status()
    return response


def remaining_timeout(deadline: float) -> float:
    remaining = deadline - time.monotonic()
    if remaining <= 0:
        raise TimeoutError("Conversation deadline reached")
    return min(HTTP_TIMEOUT_SECONDS, remaining)


def poll_next_message(client: requests.Session, endpoint: str, deadline: float) -> dict:
    while True:
        data = voxli_request(
            client, "POST", endpoint, timeout=remaining_timeout(deadline)
        ).json()
        remaining_timeout(deadline)
        if data["ready"]:
            return data
        time.sleep(min(POLL_INTERVAL_SECONDS, remaining_timeout(deadline)))


def run_test(test_result_id: str, client: requests.Session, base_url: str) -> None:
    endpoint = f"{base_url}/test-results/{test_result_id}"
    deadline = time.monotonic() + SESSION_DEADLINE_SECONDS
    for turn_number in range(MAX_TURNS + 1):
        turn = poll_next_message(client, f"{endpoint}/next-message", deadline)
        if turn.get("end_chat"):
            print(f"Test {test_result_id}: conversation ended; check scoring in Voxli")
            return
        if turn_number == MAX_TURNS:
            raise RuntimeError(f"Conversation exceeded {MAX_TURNS} turns")

        entries = get_agent_response(test_result_id, turn)
        if not entries:
            raise ValueError("Bot returned no messages or events")
        for entry in entries:
            voxli_request(
                client,
                "POST",
                f"{endpoint}/conversation",
                entry,
                timeout=remaining_timeout(deadline),
            )


def main() -> int:
    try:
        # Bot env files must not supply CLI/workflow inputs.
        api_key = os.environ["VOXLI_API_TOKEN"]
        test_result_ids = json.loads(os.environ["TEST_RESULT_IDS"])
        base_url = os.getenv("VOXLI_API_URL", "https://api.voxli.io").rstrip("/")
        load_bot_settings()
        if not api_key or not base_url:
            raise ValueError("VOXLI_API_TOKEN and VOXLI_API_URL must not be empty")
        if not isinstance(test_result_ids, list) or not test_result_ids:
            raise ValueError("TEST_RESULT_IDS must be a nonempty JSON array")
        if any(
            not isinstance(item, str) or not item.strip() for item in test_result_ids
        ):
            raise ValueError("TEST_RESULT_IDS must contain nonempty strings")
    except (KeyError, ValueError, OSError) as error:
        print(f"Bridge configuration failed: {error}")
        return 1

    failed = False
    with requests.Session() as client:
        client.headers["Authorization"] = f"Bearer {api_key}"
        for test_result_id in test_result_ids:
            try:
                run_test(test_result_id, client, base_url)
            except Exception as error:
                failed = True
                print(f"Test {test_result_id}: bridge failed: {error}")
                try:
                    voxli_request(
                        client,
                        "PATCH",
                        f"{base_url}/test-results/{test_result_id}",
                        {"status": "failed"},
                    )
                except Exception as status_error:
                    print(
                        f"Test {test_result_id}: could not mark failed: {status_error}"
                    )
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

The script polls for a tester turn and posts the entries returned by the adapter. If your bot greets first, create its session and record the greeting before the first poll. The sample uses automatic completion, so scoring follows the conversation. For late traces, use manual completion.

HTTP errors, adapter failures, deadlines, and turn limits produce a nonzero exit. The script tries to mark the affected test failed, then attempts the remaining assigned tests.

Report latency

Start with the time from sending a tester message until the bot call returns. Measure it around the bot call with time.monotonic() and convert seconds to milliseconds. If the platform already reports that measurement, use it directly.

Add the measured number to the reply’s metadata.metrics.latency_ms. The bridge preserves this metadata. For example, this entry uses an illustrative latency of 850 milliseconds:

{
  "type": "message",
  "content": "How can I help?",
  "metadata": {"metrics": {"latency_ms": 850}}
}

If one bot call produces several replies, attach its latency only to the first message. Omit the metric when it is unavailable; zero would count as a measurement.

In Settings → Metrics, create a metric with Key latency_ms, Label “Response time”, Unit Milliseconds, and Aggregates avg. Choose where to show it in Layout. The key must match exactly. See Custom Metrics for the settings and How Metrics Are Summarized for aggregation.

Configure bot settings

Put bot credentials, its endpoint, and options in .env beside the bridge. Track a placeholder .env.example with the names your adapter needs. For example:

BOT_API_URL=https://your-bot.example/chat
BOT_API_KEY=replace-locally
BOT_MODEL=your-model

Add .env and bots/*.env to .gitignore. Never put VOXLI_API_TOKEN or TEST_RESULT_IDS in a bot env file. The sample reads them from the environment before loading bot settings. Real environment variables take precedence over the file. Reuse the project’s settings loader if it already has one.

Setting	Locally	In GitHub Actions
Bot credentials	Gitignored `.env`	Secrets
Bot endpoint and plain options	`.env` or environment	Variables
`VOXLI_API_TOKEN`	Injected by the CLI	Secret containing a Voxli API key
`TEST_RESULT_IDS`	Injected by the CLI	Workflow input `test_result_ids`

The bridge defaults to https://api.voxli.io. To target another Voxli environment, set VOXLI_API_URL before voxli auth and voxli listen. That API controls the login-app redirect, and the bridge inherits the same API URL.

For several bots, use a named listener for each bot and reuse the project’s bot selector, such as --bot support. The sample selects a separate bots/<name>.env file with BOT_ENV, relative to the command’s working directory. --name identifies the agent; it does not select the file.

BOT_ENV=bots/support.env voxli listen --name support --command "python run_test.py"

Start listening

In your terminal, from the bridge’s directory and Python environment:

npm install -g @voxli/cli
voxli auth
voxli listen --name my-bot --command "python run_test.py"

voxli auth opens the browser to link your machine to your account. The CLI saves your credentials locally and injects them into the bridge. If an assistant is helping you, start the listener yourself and confirm it is running. Keep the terminal open.

Verify one test

Select the local agent named by --name and run one test. If the bot uses tools, choose a test that exercises a known tool. Wait for scoring, then read the conversation and confirm the bot’s replies and tool records are present. A failed assertion can still demonstrate that the bridge works; inspect the failure to decide whether the bot or the test needs a change.

When using the MCP, select the agent with list_agents, start the test with run_tests, poll get_run_group with status_only: true, then read the overview and get_run. Collect each repeat’s result_id and pass them to get_test_results as result_ids, with include_conversation_types: ["message", "tool", "internal-event", "public-event", "action"] to inspect full replies and tool metadata. The MCP Server serves the full setup procedure through get_voxli_guide("local-testing-setup").

Troubleshoot

Agent missing or offline: keep the listener running and check the Voxli workspace. Five minutes without a heartbeat makes a local agent offline.
Authentication fails: run voxli auth again. Remove a stale VOXLI_API_TOKEN environment override first if present.
Bridge exits before the conversation ends: the sample marks the affected test failed when it can. If the process was killed or the status update failed, cancel any unfinished work in Voxli. Fix the reported error before rerunning.
Polling never becomes ready: interactive tests may be waiting for a person. Check the run status and last reply for automated tests. Keep the bridge’s deadline in place.

To run the same bridge in CI, follow GitHub.