Evaluating a voice agent before it goes live

A voice agent demos well. That is the problem with them.

A scripted call in a quiet room, one speaker, no interruptions, a question the agent was built to answer — that call works on the first afternoon of the project. It tells you almost nothing about the call that arrives at 02:00 from a lorry cab with the window down, where the caller talks over the greeting and abandons half a sentence to read a reference number off a phone screen.

So we do not ship on a demo. We ship on a suite. Here is what is in it.

Score the transcript separately from the audio

The first useful split is between what the agent decided and how it sounded. They fail independently and they are fixed by different people.

A transcript-level test replays a conversation as text and asserts the agent's decisions: which tools it called, with what arguments, what it said, when it handed over. These are fast, they are deterministic enough to gate a pull request on, and they catch the majority of regressions — because most regressions are reasoning regressions.

An audio-level test runs real speech through the whole stack. These are slow, they are noisy, and they catch the things a transcript cannot: a name the recogniser mangles, a barge-in that arrives during synthesis, a silence the agent interprets as the end of a turn.

You need both. Teams that only build the second kind end up with a suite too slow to run, and stop running it.

The four things worth scoring

1 · Task completion

Did the caller get what they rang for? Not "did the agent respond appropriately" — did the booking change, did the status get read out, did the escalation reach a person with the context attached.

This has to be scored against an explicit outcome per test case, defined before the agent is built. If the outcome is written afterwards it will be written to match what the agent already does.

2 · Handover quality

Every voice agent hands over. The interesting question is what arrives with the caller.

We score handovers on whether the transcript, the caller's identity and the tool calls already made are attached, and whether the agent handed over at the right moment. Handing over too late is the expensive failure: the caller has already repeated themselves twice.

3 · Turn-taking latency

The number that decides whether the thing feels like a conversation. Measured end to end — from the caller stopping speaking to audio coming back — and not at the model call, which is the measurement that flatters.

It is a budget, and it is spent across the whole hop:

1. Endpointing — deciding the caller has actually stopped, rather than paused. Tuning this aggressively buys latency and costs interruptions.

2. Final transcription — the gap between the last partial and a committed transcript.
3. The agent turn — retrieval, tool calls, generation. The only part most teams measure.

4. Speech synthesis to first audio — time to the first byte of audio, not to the full utterance.

5. Network and telephony — the carrier hop, which you do not control and must still count.

4 · Interruption recovery

Callers interrupt. On a phone line that is ordinary conversational behaviour, not an error, and an agent that treats it as one talks over people.

The test is specific: start synthesis, inject caller speech partway through, and assert that synthesis stops, the partial utterance is retained, and the agent resumes from what it had rather than restarting its turn. This is worth a dedicated set of cases, because it is the failure most likely to make a caller hang up.

Build the evaluation set from real calls

Synthetic test cases encode what you imagined. Recorded calls encode what happens.

Where recordings exist and consent allows it, the evaluation set is drawn from them — including, deliberately, the calls that went badly. Where they do not exist yet, the set starts synthetic and is replaced as real traffic arrives. Either way the set grows: every production failure becomes a case, which is what stops the same bug shipping twice.

# One case. The assertions are about the decision, not the wording.
case = {
    "audio": "calls/0042-amend-booking.wav",
    "expect_tools": [("lookup_booking", {"reference": "GL-88213"}),
                     ("amend_booking", {"window": "2026-09-12T08:00/10:00"})],
    "expect_outcome": "amended",
    "expect_handover": False,
    "budget_ms": {"p50": 900, "p95": 1800},
}

Note what is not asserted: the exact sentence the agent says. Asserting phrasing makes the suite fail on every prompt change and teaches everyone to ignore it.

Gate releases on the suite

An evaluation suite that is not a gate is a dashboard, and dashboards get ignored under deadline.

Ours runs on every change to the agent, and a release requires the transcript-level suite green and the audio-level suite within its thresholds. When a threshold is relaxed, the relaxation is a reviewed change with a reason, not a quiet edit.

What we do not measure

We do not score sentiment. We do not score whether callers realised they were talking to software — they are told in the first sentence, because a caller who does not know cannot consent to being recorded.

And we do not report a single accuracy number. A voice agent is four systems in a trench coat, and one number averaged across them tells you which afternoon to waste.