How to tell if your AI voice agent is actually working
You cannot hear every call it takes. Here is what to measure, how to catch what it gets wrong, and the dashboard that turns a nervous go-live into a number you can manage.
How do you know if an AI voice agent is working?
Measure outcomes, not uptime. Track the share of calls the AI resolves without a person, how cleanly it hands over the rest, what happens after hours and why people are calling. Score every call automatically against criteria you define, review the failures every week, and treat each one as a fault to log, fix, retest and monitor.
Why is it so hard to know if an AI voice agent is working?
Because the calls happen without you, the callers rarely complain to the AI, and the vendor’s dashboard measures the wrong thing.
Anyone who has switched on an AI receptionist knows the feeling. The first week is spent wondering what it is saying to people. Is it answering properly? Did the caller who hung up get what they needed? Is the routing right at 6pm on a Friday? Your test calls went well, but you made those calls, you knew what to say, and you were polite.
Real callers are none of those things. They interrupt, call from cars, ask two questions at once and switch language halfway through a sentence. When the AI gets it wrong, most of them do not tell you. They hang up and call a competitor, or they call back and complain to a person, days later, about something that happened on a call nobody heard.
Most platform dashboards will not catch it either. They report minutes used, calls answered and uptime. A call that was answered and went badly counts the same as one that went perfectly.
What changes when the receptionist is software?
A person’s mistake is a training conversation. An AI’s mistake is a bug, and it repeats on every call until someone fixes it.
When an inexperienced receptionist mishandles a call, the fix is quick and human: a more experienced colleague takes over, has a word, and the next call goes better. Nobody writes a report. With an AI agent, the same mistake is a system fault. It will happen to every caller who asks that question, at every hour, until the instructions, the routing or the data behind it are changed. And the change can break something else.
| When something goes wrong | Human receptionist | AI voice agent |
|---|---|---|
| Who notices | A colleague nearby, usually straight away | Nobody, unless calls are scored or reviewed |
| How often it repeats | Rarely, once corrected | On every similar call until it is fixed |
| How it is fixed | A quick word or a handover | A change to instructions, routing or data |
| Risk of the fix | Low | The change can break other call types |
| How you know it worked | You hear the next call | You retest and watch the numbers |
That is why an AI voice agent needs the same discipline as any other production system: a fault log, an owner for each fault, and a loop that does not close until the fix has been proven on real calls.
The fault loop: how an AI voice problem gets fixed and stays fixed
What should you measure to know an AI voice agent is working?
Five numbers. Together they tell you whether callers are being served, not just answered.
| Measure | What it tells you | Warning sign |
|---|---|---|
| Resolution rate | Share of calls the AI completes with no person involved | Falls after a change, or never rises after launch |
| Handover quality | How fast the AI passes on what it cannot handle, and whether the caller reaches someone | Long calls before a transfer; callers who hang up after it |
| Failure rate | Calls that dropped, errored, looped or ended with the caller’s need unmet | Any spike, especially on one call type or one time of day |
| After-hours outcomes | What happened to calls when nobody was in the office | A different result from office hours, or nobody reviewing it |
| Call reasons | Why people call, ranked | A reason the AI was never set up for, growing week on week |
Notice what is missing: minutes used and calls answered. Those are billing figures. They matter for what the agent costs, not for whether it works.
A fast handover is a good result, not a failure. In one live deployment we measured, the AI resolved 38% of engaged calls on its own, and when it could not help, the median call reached a person in 42 seconds. Those callers were not trapped in a loop. The AI recognised quickly that a person was needed, which is exactly what it should do. The full call analysis shows how those numbers break down.
Why is testing before go-live not enough?
Because test calls are polite and predictable, and every change you make afterwards can move the results.
Pre-launch testing proves the agent can handle the calls you thought of. It cannot prove how it handles the ones you did not. And an AI voice agent is never really finished: you will adjust its instructions, add an answer, change a transfer rule. Each change is a new version of the receptionist.
In our measured deployment, a single reconfiguration moved the share of calls handed to a person by 13 percentage points, and the new level held for weeks. Call volume rose over the same period, so the change was not the only factor. But the lesson stands: one edit can shift outcomes for every caller, and you will only see it if you are watching the numbers before and after.
How do you check what happens to calls after hours?
Assume after hours is where problems hide, because it is the one time nobody is there to notice.
An AI receptionist is often bought for the evenings, weekends and public holidays. Those are also the hours with the least supervision and the most complicated rules: different greetings, different routing, an emergency number, a December shutdown. A routing schedule that is wrong on a Saturday can be wrong for months before anyone calls on a Saturday and complains.
- Test the schedule end to end, including public holidays, before you rely on it.
- Report after-hours calls separately, because the mix of callers and reasons is different.
- Review them the next working morning, so a missed urgent call is caught within hours, not weeks.
A receptionist’s mistake is a training conversation. An AI’s mistake is a fault to log, fix and prove.
WhichVoIP editorial viewWhat should an AI voice performance dashboard show?
One score the business can understand at a glance, and the reasons behind it.
The business owner does not want transcripts. They want to know whether calls are being served, how many were successful against how many were not, and what is going wrong. The most useful format we have used is a single performance score built from every call, with the outcome split and the call reasons underneath it.
Example AI voice performance dashboard (illustrative figures)
How the score works. Every call is scored automatically against criteria written in plain language: did the caller get what they needed, was the handover clean, was the information correct. Most serious AI voice platforms can now do this; ElevenLabs, for example, marks each call as a success, failure or unknown against your own criteria and explains why. Combine those results into one score, weighted towards the outcomes your business cares about, and track it month on month.
The score is not the point on its own. It earns trust because the owner can click from it to the failed calls behind it, and see that each one has been logged and is being fixed.
Why are call reasons the most valuable thing an AI agent records?
Because for the first time, every call is categorised, and that tells a business what its customers actually need.
A human receptionist answers a hundred calls a week and remembers a handful. An AI agent records why every caller rang. Put that in a ranked list and the business sees patterns no one had time to notice before.
Questions the website should answer
- ExampleHours, directions
- ActionFix the website
Demand you can sell into
- ExampleBooking requests
- ActionOnline booking
Problems before they spread
- ExampleRepeat complaints
- ActionFix the cause
This is the value that outlasts the novelty. The AI answers the phone, but the call-reason report changes how the business runs.
How often should you review an AI voice agent?
Daily at first, then weekly, and always after a change.
Daily. Read or listen to every failed call and every handover. This is when the gap between testing and real callers shows up.
Failure review. Group the week’s failures by cause, log each with an owner, and decide what to fix next.
Retest and compare. Run your set of real call types, then compare the next week’s numbers with the week before.
Owner report. The performance score, the outcome split, after-hours results and the top call reasons, with what was fixed.
Our verdict
An AI voice agent is not a project you finish. It is a receptionist that never gets tired and never learns on its own, so somebody has to do the learning for it. The businesses that get value from AI voice measure outcomes, treat every bad call as a fault with an owner, retest every change against real calls, and give the owner one score and one list of call reasons each month. Those that rely on uptime and test calls find out about problems from their customers.
How well is yours being watched?
Answer 15 quick questions and get a health score for how you monitor your AI voice agent, with the gaps to close first.
Frequently asked questions
How do you measure the performance of an AI voice agent?
What is a good resolution rate for an AI receptionist?
How can I tell if my AI voice agent is failing calls?
Is testing an AI voice agent before go-live enough?
What should an AI receptionist report to the business owner?
Why do call reasons matter?
Keep reading
Sources: Resolution, handover and configuration figures are from WhichVoIP’s analysis of 2,544 calls to a live AI receptionist we operate (2,151 engaged calls; greeting hang-ups excluded). Call scoring capability from ElevenLabs Agents platform documentation, success evaluation (accessed 1 October 2026). Dashboard figures are illustrative. Verified 1 October 2026.