Voxli Voxli

GitHub Integration

Run Voxli tests directly from GitHub Actions. When you trigger a test run, Voxli dispatches your workflow with the test details—your workflow handles the conversation loop.

How It Works

1. Trigger: When you run tests with a GitHub agent, Voxli triggers your workflow via workflow_dispatch.

2. Run: Your workflow runs and iterates over each test result ID.

3. Loop: The script relays messages between Voxli and your agent until the conversation should end.

Prerequisites

Before setting up a GitHub agent:

  • Install the Voxli GitHub App via Settings → Integrations → Add Integration → GitHub
  • Create a workflow file in your repository’s .github/workflows/ directory
  • Add your API token as a repository secret named VOXLI_API_TOKEN

Workflow Setup

Your workflow needs the workflow_dispatch trigger so that Voxli can trigger it. Voxli will pass test_result_ids as a JSON array input to it.

name: Run Voxli Test


on:
  workflow_dispatch:
    inputs:
      test_result_ids:
        description: "JSON array of test result IDs"
        required: true


jobs:
  run-test:
    runs-on: ubuntu-latest
    # Interactive sessions wait on a human between turns, so this job can idle
    # for a while. Cap it so a session that never ends cannot bill runner
    # minutes indefinitely - self-hosted runners have no ceiling of their own.
    timeout-minutes: 120
    steps:
      - uses: actions/checkout@v4


      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"


      - name: Install dependencies
        run: pip install requests


      - name: Run test
        env:
          VOXLI_API_TOKEN: ${{ secrets.VOXLI_API_TOKEN }}
          VOXLI_TEST_RESULT_IDS: ${{ github.event.inputs.test_result_ids }}
        run: python run_test.py

Running the Conversation

The workflow runs a Python script that handles the conversation loop. This script receives the test_result_ids from the workflow input and runs each test:

"""
Run Voxli test conversations.


Environment variables:
- VOXLI_API_TOKEN: Your API key (required)
- VOXLI_TEST_RESULT_IDS: JSON array of test result IDs (required)
"""


import json
import os
import sys
import time
import requests




def get_agent_response(message: str) -> str:
    """
    Replace this with your actual agent integration.


    Receives a message from the Voxli tester and returns
    your chatbot's response.
    """
    return "The meaning of life is 42."




# In interactive sessions the tester is a person (or agent) typing, so `ready`
# can stay false for minutes between turns — far longer than an automated run.
# Voxli ends abandoned sessions on its side, but don't rely on that alone: this
# runner is a billed CI job, so it holds its own deadline and turn cap and exits
# on its own if the server never says end_chat.
SESSION_DEADLINE_SECONDS = 90 * 60
MAX_TURNS = 200
POLL_INTERVAL_SECONDS = 2




def poll_next_message(endpoint: str, headers: dict, deadline: float) -> str | None:
    """Wait for the next tester message, or None once the chat is over.


    `deadline` is an absolute time covering the whole session, not one turn,
    so a long chat cannot extend the job indefinitely one turn at a time.
    """
    while True:
        response = requests.post(endpoint, headers=headers)
        response.raise_for_status()
        data = response.json()
        if data["ready"]:
            return None if data.get("end_chat") else data["message"]
        if time.time() > deadline:
            raise TimeoutError("Session deadline reached waiting for a message")
        time.sleep(POLL_INTERVAL_SECONDS)




def run_test(test_result_id: str, api_key: str, base_url: str):
    headers = {"Authorization": f"Bearer {api_key}"}
    conversation_endpoint = f"{base_url}/test-results/{test_result_id}/conversation"
    generate_endpoint = f"{base_url}/test-results/{test_result_id}/next-message"
    deadline = time.time() + SESSION_DEADLINE_SECONDS


    # Get first message from Voxli
    tester_message = poll_next_message(generate_endpoint, headers, deadline)


    # Converse until Voxli ends the chat (end_chat=true), bounded by our own
    # deadline and turn cap so the job always terminates.
    for _ in range(MAX_TURNS):
        if tester_message is None:
            print(f"Test {test_result_id} completed")
            return


        agent_response = get_agent_response(tester_message)


        # Record agent response
        response = requests.post(
            conversation_endpoint, headers=headers, json={"type": "message", "content": agent_response}
        )
        response.raise_for_status()


        # Get next message from Voxli
        tester_message = poll_next_message(generate_endpoint, headers, deadline)


    print(f"Test {test_result_id} stopped after {MAX_TURNS} turns")




def main():
    api_key = os.getenv("VOXLI_API_TOKEN")
    test_result_ids_raw = os.getenv("VOXLI_TEST_RESULT_IDS")
    base_url = os.getenv("VOXLI_API_URL", "https://api.voxli.io")


    if not api_key or not test_result_ids_raw:
        print("Error: VOXLI_API_TOKEN and VOXLI_TEST_RESULT_IDS are required")
        sys.exit(1)


    test_result_ids = json.loads(test_result_ids_raw)


    for test_result_id in test_result_ids:
        run_test(test_result_id, api_key, base_url)




if __name__ == "__main__":
    main()

Environment

Local