The Customer Who Gave Up Counts as Contained
Why the most popular AI customer service metric can hide failure, and what leaders should measure instead. Containment rates make AI customer service look like a success even when customers leave unhelped. What analysts, one well-known company’s reversal and customer data say, a better scoreboard, and six decisions for the next quarter.
In brief
- Many AI customer service programs are judged on containment or deflection, which count conversations that never reached a person. Neither tells you whether the problem was solved.
- A customer who gives up is counted the same as a customer who is delighted.
- Klarna’s CEO said publicly in 2025 that cost had become too dominant a factor and quality suffered. Gartner forecasts that half of organizations planning large service workforce cuts will abandon them by 2027.
- Gartner also projects that generative AI’s cost per resolution will exceed many offshore human agent costs by 2030. That is a forecast, not a fact, but it changes how to think about “cheaper.”
- The practical fix is a better scoreboard: verified resolution, repeat contact, time to a human, and fully loaded cost per resolved issue.
1. A tale of two customers
The following is an illustration, not a documented case.
Two customers contact a company about a billing error. The first describes the problem, the AI agent explains the correction, the account is fixed, and the customer logs off. The second describes the same problem and gets a generic answer. They rephrase it, get another generic answer, and eventually close the window and call the next day.
On the dashboard, these two customers look identical. Both conversations ended without reaching a human. Both are counted as “contained,” and both are credited as AI successes. One was a success. The other was a failure that the metric recorded as a win, and the company will now pay for it twice: once in a lost customer’s goodwill and once in the second contact.
That is the problem with the metric in a single sentence. It measures where a conversation ended, not whether anything was solved.
2. A ladder of metrics
The industry uses several overlapping terms, and they are not interchangeable. A vendor’s glossary describes a four-step ladder, which we find a useful way to organize them. Because the vendor sells a resolution-based product, the framing deserves a caveat, and we use it as a vocabulary, not as evidence.
Deflection rate is the share of inbound queries that never reach a human. The glossary says it “tells you nothing about customer outcomes.”
Containment rate is the share of AI conversations that end without a handoff. It cannot distinguish a satisfied customer from one who gave up.
Automated resolution rate requires the conversation to be relevant, accurate and contained. The same source notes that most vendors use their own AI to grade this, which means the system is marking its own homework.
Verified resolution combines automated tracking with independent quality scoring across all conversations, and treats resolution and reopen rate as a pair. The glossary says a high resolution rate with a high reopen rate is “functionally, a containment rate wearing better clothes.”
The vendor also publishes performance figures, including a 76% average resolution rate across customers. These are self-reported with no independent audit or sampling method described. We do not use them as a benchmark here, and we would encourage leaders to be equally careful with any vendor’s own numbers.
The framework is useful regardless of who published it. As we see it, each step up the ladder moves the question from “did the conversation stop” toward “did the customer’s problem go away.”
3. When cost led, quality followed: the Klarna example
The most public example of the tension is Klarna. In 2024, the company said its AI assistant could do the work of 700 agents and handle 75% of chats in more than 35 languages, and it described plans to shrink from 5,000 to 2,000 employees through attrition.
In May 2025, Klarna CEO Sebastian Siemiatkowski told Bloomberg that the push toward automation “went too far.” He said “cost unfortunately seems to have been a too predominant evaluation factor” and that this led to lower quality. He added that “Really investing in the quality of the human support is the way of the future for us” and that customers should know “there will be always a human if you want.” The company began piloting a model with remote human support staff, while remaining, in the article’s words, bullish on AI and building a digital financial assistant.
We would add three cautions. This is one company, so it is a case study, not a trend. The original claims were Klarna’s own, and the later reassessment is the CEO’s own words as reported by a technology news site citing Bloomberg, so we have not seen independent customer data on what changed. And the same reports describe a company under financial pressure, with a valuation that had fallen sharply and IPO plans reportedly paused, so there may have been other reasons for the shift.
Still, the sequence is instructive. A company optimized for a measurable number, cost per contact, and later concluded that the number had crowded out the thing customers cared about.
4. What analysts forecast
Gartner has published two predictions worth reading together. Both are forecasts from an analyst firm, not measurements, and the firm sells research and advisory services, but they reflect what its customer service practice is hearing from leaders.
Workforce plans. In June 2025, Gartner predicted that by 2027, 50% of organizations that planned to significantly cut their customer service workforce will abandon those plans. The reasoning was that many companies find it hard to reach their “agent-less” staffing goals. A Gartner poll of 163 customer service and support leaders in March 2025 found that 95% plan to keep human agents to help define AI’s role. Gartner’s analyst Kathy Ross said that “a hybrid approach, where AI and human agents work in tandem, is the most effective strategy,” and the release describes AI as “not a panacea.”
Cost. In January 2026, Gartner predicted that by 2030, the cost per resolution for generative AI in customer service will exceed $3, which it says is higher than many B2C offshore human agents. It names three drivers: rising data center costs, AI vendors shifting from subsidized growth toward profitability, and more complex use cases that consume more tokens and need expensive talent. Senior Director Analyst Patrick Quinlan said: “Full automation will be prohibitively expensive for most organizations.” The release gives no current cost baseline, so we cannot say how much costs would need to rise.
The same release predicts that by 2028, regulations requiring access to a human will raise assisted service volume by 30%, and that customers will opt out of AI by default. Treat that as a signal of analyst expectations, not as an established outcome.
The practical takeaway is not that AI is a bad investment. Gartner’s own recommendation is to use AI to drive engagement, not mainly to cut costs, and to keep staffing adequate so customers do not wait long for a person.
5. What customers say
Gartner’s August 2026 survey of 3,566 customers, fielded in February and March, adds the demand side. It is Gartner’s own release, reposted by Insurance-Canada.ca, and it does not describe the exact question wording, so we treat it as an analyst survey.
- 87% said it is “essential for companies to provide an option to reach a human agent when using GenAI.”
- 50% said their interactions are easier when companies use GenAI.
- Customers were about three times more likely to use a third-party GenAI tool, such as ChatGPT, Gemini or Copilot, than a company-provided chatbot in their most recent service interaction.
- Among customers unwilling to engage with AI, the most common thing that could change their minds was the ability to switch to a human. Gartner advises against making GenAI a mandatory first step, because customers forced through multiple unsuccessful AI interactions are less likely to use the tool again.
That last point connects directly to the metric problem. If a customer who gave up is counted as contained, the company is rewarded for exactly the experience that makes the customer less likely to return to the channel. The metric can improve while the thing it is meant to proxy gets worse.
The finding that customers often reach for outside AI tools instead of company chatbots is a separate warning. It suggests that a company’s own agent is competing with free alternatives for the customer’s first attempt.
6. A better scoreboard
None of the metrics below is new. The shift is in making them the headline and treating containment as a diagnostic, not a goal.
Verified resolution rate. The share of conversations where the customer’s issue was actually solved, confirmed by something other than the AI’s own judgment: an explicit customer signal, a human reviewer on a sample, or the absence of a repeat contact. Define the evidence you accept in writing.
Repeat contact rate. The share of customers who contact you again about the same issue within a set window, such as seven days, across any channel. This is the most direct way to expose a customer who gave up and called the next day. It requires linking contacts by customer and topic, which is a data task worth the effort.
Abandonment and hang-up rate. The share of conversations where the customer left before completion. In a voice agent, this is often visible in the call data.
Escalation quality. When the AI hands off, was it timely, did the person receive context, and did the customer have to repeat themselves?
Time to a human. How long it takes to reach a person when one is requested. It is a customer experience measure, and as a legal matter in some places it is becoming a compliance measure too.
Fully loaded cost per resolved issue. Not cost per conversation, and not cost per AI interaction. Add the AI platform cost, the human cost of every escalation and every repeat contact, and any quality assurance effort, and divide by issues that were actually resolved. This is the number Gartner’s cost forecast is about, and it is the one most likely to move as AI prices change.
Complexity mix. Resolution rates mean little unless you know what types of questions the AI is being asked. A high rate on password resets says little about performance on disputes.
Independent quality sampling. A regular sample of conversations reviewed by someone who is not the system that produced them, for accuracy, tone and completeness.
7. A worked example with illustrative numbers
The following numbers are invented to show the arithmetic. They are not from any source.
Suppose an AI agent handles 1,000 conversations in a month. The dashboard shows 700 ended without a handoff, so containment is 70%.
Now look underneath. Of those 700, a review finds that 120 customers left without getting an answer to their question. Another 130 contacted the company again within seven days about the same issue. That leaves 450 conversations where the customer’s problem was solved and stayed solved.
Verified resolution is 45%, not 70%.
The 250 failures did not disappear. Some of them returned as repeat contacts, which add to human workload. The company’s “cost per contained conversation” looks low, but its fully loaded cost per resolved issue is higher, because the human cost of cleaning up after the 250 is attributed to nobody’s dashboard.
The point is not that a 25-point gap is typical. We do not know what typical is. It is that the gap exists by construction, and the only way to find yours is to measure it.
8. Questions to ask your own data
- Of the conversations we count as contained, what fraction ended with the customer leaving without an answer?
- How many customers contact us again within seven days, and how many of those had just spoken to the AI?
- Which question types have the lowest verified resolution, and are those the questions that matter most to the business?
- When customers ask for a person, how long does it take, and what do they have to repeat?
- How is our cost per resolved issue trending as volume and complexity change?
- Who reviews a sample of conversations independently, and what do they find?
9. Objections we hear, and our answers
“Customer surveys have low response rates, so we cannot measure satisfaction.” Survey response rates are low, and one vendor’s glossary puts typical CSAT response at roughly 2% to 8% of conversations. That is a reason to supplement surveys with behavioral measures such as repeat contact, not a reason to rely on containment.
“Containment is simple and everyone uses it.” Simplicity is its appeal and its weakness. Many leaders use it as a diagnostic alongside other measures. The risk is making it the target.
“Our vendor already reports resolution.” Ask how it is graded. If the vendor’s own AI marks the outcome, you have a self-assessment. An independent sample is the check.
“We will measure more once the program matures.” The measurement decisions shape the program. A system optimized for containment will get better at containment.
“A bot that is wrong cheaply still saves money.” Possibly, until you count the repeat contact, the lost customer and the possibility that the customer switches to someone else’s tool.
10. Six decisions for the next quarter
- Make verified resolution the headline metric. Keep containment as a diagnostic.
- Link contacts. Build the ability to see repeat contact by customer and topic across channels.
- Sample independently. Put a regular, human review of conversations in someone’s job.
- Cost it fully. Report cost per resolved issue, including human escalations and repeat contacts.
- Segment by complexity. Report results by question type, not only in aggregate.
- Keep a human path visible. Treat the route to a person as part of the product, and track how it performs.
What we do not know
- How large the gap between containment and verified resolution typically is. We found vendor figures and illustrative examples, but no independent benchmark we would trust.
- Whether Klarna’s experience is common, and what its customer data showed. We have only the CEO’s account as reported.
- Whether Gartner’s cost-per-resolution forecast will hold, given that the release gives no current baseline and depends on assumptions about AI pricing.
- How AI pricing will actually move, and how efficiency gains offset the cost drivers Gartner names.
- Whether customers’ stated preference for a human option, reported by analysts, translates into behavior.
- How repeat contact should best be measured across channels, which depends on a company’s data.
This article is general information and not advice. We relied on analyst releases, trade and press reports and a vendor glossary, and none of the figures cited here are from our own measurement.
See it working
Not sure how your AI conversations are really performing? GoZupees works with teams to track quality, handoff and outcomes. If that is on your roadmap, book a demo at www.gozupees.com.