AI agents are being deployed faster than CX can figure out how to measure their performance. It's a major blind spot, and the consequences are real.
More than one-third of consumers in the US told a Liveops survey they would lose trust in a brand if its automated or AI-driven customer service failed to handle an issue – even if a human agent later stepped in to resolve the issue.
However, consumer trust in AI more than doubles when a human option is visible. In fact, 55 percent of consumers trust AI to resolve their issue when a clear option to reach a human is available. When there's no human option, that figure drops to 26 percent, according to research published by Five9 in October 2026.
The practitioner's perspective is simple: you made the investment, integrated the capabilities, trained the team. Now, the big question is "What should I track and what does good look like?"
Because AI agents work autonomously and execute workflows, rather than single tasks, their performance management is complex. Errors can take time to surface and consumer sentiment must be captured at multiple points in the journey.
This means organizations need to understand and implement a range of metrics and align them with the wider objectives of the business to ensure not only that the AI agents are delivering as intended, but that they remain aligned with the strategic goals they were deployed to drive.
This article explores:
- The differences between measuring the performance of AI agents, software and human agents.
- The key CX and business metrics that can be used to create a performance management framework.
- How to set baseline targets and avoid common errors.
- And what to do if underperformance is detected.
Why measuring AI agents is different from measuring traditional software – or humans
Traditional software is deterministic. It executes fixed, rigid lines of code to guarantee the exact same result every single time. It's easy to track how its performance meets expectations or delivers benefits, for example through volume of tasks completed, time taken, total uptime or speed.
Because AI agents act autonomously within set guardrails and using specific data and criteria to complete tasks, they are non-deterministic. They can produce different outputs, reasoning paths, or action sequences in response to the same request or prompt. The metrics that usually measure software do not tell the full story when applied to these scenarios.
When measuring the performance of humans, front-line workers are expected to demonstrate empathy, respond with speed, share accurate information and ensure customers leave the interaction satisfied. Common metrics include first call resolution (FCR), average handle time (AHT), or tickets solved per hour.
AI agents fall between the two. They require metrics that can capture technical performance – for example, speed, uptime, resolution and volume – and they are also measured against the same metrics as humans, to provide a like-for-like comparison for service leaders.
Forward-thinking organizations may also add a third dimension, customer sentiment. For example, by capturing NPS or satisfaction with the AI agent.
However, measuring and reporting metrics in isolation does not capture the full picture. Instead, a framework is needed.
The core framework of performance management for AI agents
The first step is establishing the AI agent's goals, then deciding which metrics are most appropriate for measuring its attainment of those goals.
"No single number gives you the full picture," says Sudarshan Reddy Gurampatti, senior machine learning engineer and AI architect with JPMorgan Chase & Co. "The useful part is looking at these signals together instead of relying on whichever metric looks best."
Task success and accuracy
If an AI agent cannot successfully and accurately handle a customer contact, it should probably not be used. Five9's research found 92 percent of consumers give AI fewer than 10 minutes before asking for a human. Among those who switch, 46 percent give the same reason: the bot repeatedly misunderstood them.
To measure accuracy and success, practitioners should measure the completion rate of AI agent tasks and also analyze transcripts to detect instances of incorrect information, or analyze contact logs to detect repeat contacts, which can also point to inaccurate or incomplete case handling.
Quality and reliability
Five9's research found 57 percent of consumers believe customer service quality has improved over the last decade – a time when AI has been deployed to transform service reliability. To ensure your AI agent meets quality and reliability standards, measure consistency, hallucination or error rate, and escalation rate. These can all demonstrate the quality and reliability of your AI agent. Customer effort score is also important here.
Efficiency
Efficiency takes many forms, from operations to cost. The traditional CX metrics, such as time saved and cost per contact can all help track efficiency, however, for granular accuracy these metrics must be combined with others or tracked over time. For example, time saved should be combined with repeat contact rate. This is because a shorter initial contact that leads to multiple follow-ups has not improved efficiency – it has simply moved a task to another agent or day.
Business impact and ROI
Revenue, upsell rate, cost reduction and customer satisfaction can all be measured to indicate whether an AI agent is delivering a return and helping in the attainment of wider business goals. The CX Network report Balancing the Books on AI, explains this in detail.
Safety, risk and compliance
AI that is not adequately planned from the outset can carry financial consequences that only surface down the road. This happens when AI is deployed without the correct guardrails and testing, leading to trust erosion, operational friction, or weaker decision quality. For organizations operating in the EU, there could also be financial penalties under the EU AI Act. Key metrics to apply here include policy adherence, security incidents and auditability. It is advisable to assign responsibility of this to a dedicated person.
Putting it into practice
Measurements should start during testing to provide a clear indication of how a tool performs over time – and in real-world environments vs sandboxes. Continuous A/B testing is helpful for a service leader to track how successful an AI agent is at handling a specific scenario in comparison with a human agent. Human review must occur at specific intervals for every scenario an AI agent handles and, in line with regulatory requirements, this should also be documented.
2 Common mistakes practitioners make when measuring AI agent performance
The work of measuring AI agent performance is complex, meaning it's easy to overlook the basics. This can include only measuring at launch, ignoring costs, failing to ensure a human-in-the-loop, or measuring specific performance dimensions in isolation without the context required to give a full picture.
They don't measure if the resolution holds
Another common oversight is treating resolution rate as a fixed outcome, rather than something that can change with time.
Nixalkumar Patel, a digital commerce leader focused on AI-driven customer experience who currently serves as a senior product manager at LG Electronics, says the most appropriate metric to measure the success of AI in a service journey is Post-Resolution Rework Rate, or PRR.
"PRR is the percentage of AI-resolved customer interactions that later require corrective customer, employee or operational activity because the original resolution did not hold," he explains.
A simple calculation is: PRR = (AI-resolved cases requiring corrective activity ÷ AI-resolved cases that have completed their observation window) × 100
"The observation-window requirement matters," Patel says. "A case resolved five minutes ago should not be compared with one that has had two weeks to generate downstream issues. PRR should only include cases old enough for the expected customer outcome to become observable."
PRR is not intended to replace existing CX measures, Patel explains. Instead, it adds another view: resolution durability.
They depend too much on containment rate
Containment does not measure whether your customer's problem was solved. It measures whether they gave up trying to reach a human.
Marie Angselius Schönbeck, CMO of Teneo.ai says: "Consider what is actually happening in contained calls. Some customers genuinely resolved their issue – the AI answered their question, completed their transaction, gave them what they needed. That is a success worth measuring. Others hit a wall. They could not get to the information they needed. The containment rate cannot tell you which of those two groups is larger."
What to do if your AI agent is underperforming
If your monitoring and metrics indicate underperformance, early and decisive action is essential.
"Poor AI doesn't only frustrate customers, it creates additional work for support teams," Gurampatti says. "If agents regularly have to correct AI-generated responses, apologize for incorrect information, or work through escalations that arrive without context, the AI may be moving work downstream rather than removing it."
This is where baselines and benchmarks become particularly useful. For example, if resolution and repeat contact rate fall short of their baseline, intervention is required.
As a first step, testing logs can be checked to identify when model performance started to drift. With this information to hand it is possible to find the specific cause for the underperformance and adjust processes accordingly.
Customer insights – whether gathered actively or passively – are also a means by which to identify underperformance. Did CSat or NPS decline after an agent was introduced? Has customer effort increased?
Then there are the cases when an AI agent is wrong and a recovery plan is required.
In this article for CX Network, Hemang Upadhyay, a senior product and AI leader with LG North America, says this should not be a generic escalation path, or a manual workaround buried in the contact center. Instead, a designed recovery layer is required that "catches a failing AI interaction, explains what happened, escalates with context, repairs the customer outcome, and stops the same failure from happening again".
Case study: How Orpi measured the benefits of custom-built AI agents
When French real estate cooperative Orpi created a custom AI platform to unify its network of 1,250 agencies, it didn't track the impact through NPS, but via lead quality and productivity.
Orpi introduced a new, conversational UX in 2026 to give house hunters a new front door to their property searches. This meant that instead of simply using filters to search for three bedroom apartments in Paris under €1 million, users could instead ask for properties close to a school and public transport, or with strong rental-investment potential.
Mania Ginisty-Gold, digital and CRM manager for Orpi, says: "Because needs are already understood, leads reach Orpi's agents warmer and better-qualified."
To measure the quality of the experience delivered, Orpi wasn't just looking at CX, financial and business metrics. Ginisty-Gold says the focus was on "lead quality and conversion, engagement, and agent productivity, ultimately, how many conversations turn into viewings and [listings]".
She adds: "There's also a forward-looking angle: the goal is to connect Orpi's AI agents to the AI agents of public search engines like ChatGPT and Google's AI, where customers increasingly start their search. Because Orpi's agents are trained on its own catalog and customers, they will surface better in AI search, so part of the ROI will come from new traffic captured upstream."
Orpi president, Guillaume Martinaud, says: "The real estate journey of tomorrow will be increasingly AI-assisted, and we chose to take the lead. Our clients expect a fluid search experience capable of understanding their real needs as if they were speaking to a human."
Read this story in full on CX Network.
Gurampatti advises: "Start small. Pick one customer-facing metric – Customer Effort Score, repeat contact rate – and look at it alongside your existing automation numbers.
"For example, if containment climbs after a new AI release, check what happens to repeat contacts over the same period. Containment up and repeat contacts down is a good sign. Containment up while the same customers keep coming back means the containment number may be hiding a problem."
AI and human CX is the future – and you need a way to measure it
Commenting on Five9's research, CEO Amit Mathradas said: "This research validates how we see CX evolving: consumers aren't asking companies to choose between AI and people; they're asking for both, done right.
"That's the core of Humantic CX, our vision for AI and human agents working as one system, where AI absorbs the routine, so a person is always there for the moments that need judgment and empathy," Mathradas continued. "The data captures the impact of this approach clearly. Every interaction where we get that handoff right builds more trust for the next one, and that's smarter CX with every interaction."
However, CX Network's 2026 research into the state of CX, found 42 percent of survey respondents feel there are "a lot of CX benefits that go unmeasured" in their organizations.
Due to their complexity, AI agents risk contributing to this trend.
The business world has become adept at managing the performance of human agents and managing the performance of software. But where the two intersect has emerged as a major blind spot.
As the insights in this article demonstrate, there are steps that can be taken to ensure AI agents are not only monitored, but managed to deliver maximum results.