Three things are keeping CX leaders up at night. Testing isn't one of them.

CX Today sat down with Nicole Kyle of CMP Research at CCW Vegas to ask what's keeping CX leaders up at night, and the answer is worth reading because it's built on research across hundreds of organisations rather than vendor enthusiasm. The interview is on YouTube if you'd rather hear it.

Three things came out of it.

Customer analytics. Data is everywhere and none of it is centralised or clean. Some teams have sophisticated tooling and still can't see a complete customer journey because the data is fragmented across systems. Others are stuck with legacy vendors and no internal expertise to fix it. Kyle calls the gap between analytical ambition and operational reality one of the defining operational problems of 2026.

Self-service adoption. Consumer demand for self-service has risen 6% over the last three years. Delivery hasn't kept up. Resolution and personalisation rates are poor enough that customers try self-service, have a bad time, and stop trying.

The AI workforce transition. The people-side of AI transformation is being deprioritised under cost pressure, while the humans doing the work are genuinely anxious about their roles. Kyle's warning is that losing good staff through a badly managed transition costs more than the AI saved.

That's a fair and unsentimental list. I'd add a fourth, and it's the one that never makes these lists, because no board paper has ever had a line item called we don't know if our phone journeys work.

TL;DR

• The three named worries are all real and all downstream of the same missing measurement.
• An analytics platform tells you what happened to customers who already had a bad time. It's a lagging indicator assembled from wreckage. A test tells you the journey is broken before a customer finds out.
• The fragmented-journey problem has an unglamorous shortcut. You don't have to stitch nine systems together to see one whole journey if you can simply run the journey yourself, end to end, on demand.
• Self-service abandonment isn't a mystery to be solved with better dashboards. It's a quality problem, and quality is testable.
• Testing isn't on the list because full journey coverage used to be priced out of reach, so teams settled for five golden paths and a human dialling the number on release morning. That's the gap where outages live, and in regulated industries, where fines live.

The analytics gap has a back door

The first finding is the one I'd push hardest on, because the framing is subtly expensive.

If your problem is "I cannot see a complete customer journey," the assumed solution is a data project. Consolidate the warehouse, fix the pipelines, buy the platform, hire the expertise. That's a multi-quarter programme and Kyle is right that most organisations are stuck partway through one.

But notice what you're actually trying to reconstruct. You're taking millions of fragments of telemetry left behind by real customers and trying to infer, after the fact, what their experience was like. That's forensics. It's genuinely useful, and it is also the most expensive possible way to answer the question, because it only works once enough customers have already had the experience you were hoping to evaluate.

There's a back door. Instead of reassembling the journey from evidence, run the journey. Have something phone your actual number, behave like a real customer with real intent, go all the way through the flow, and record everything that happened. You get a complete journey view with no fragmentation, because nothing was fragmented in the first place. One caller, one call, one continuous record of what the customer heard and how long they waited for it.

That doesn't replace your analytics programme. It does mean you can answer "does the journey work" this afternoon rather than at the end of the data roadmap.

"Demand up 6%, delivery flat" is a QA finding

The self-service finding is the one that gets misdiagnosed most often. Demand is up, resolution rates are poor, customers abandon. The instinct is to treat this as a design problem or an adoption problem, so teams redesign the flow and add nudges.

Sometimes it is a design problem. Very often it's simpler than that. The journey is broken in a specific place, on a specific input, and nobody knows because nobody checked.

The bot mishears "four" as "for" when the caller is in a car. The authentication step accepts a date of birth in one format and silently fails the other. The CRM lookup times out at 1.4 seconds and the bot fills the silence by repeating itself. The handoff to a human works from the main menu and dead-ends from the billing sub-menu. The new TTS voice reads your reference numbers wrong. None of these show up as a design flaw. All of them show up as an abandoned self-service session and a customer who has now learned not to bother.

Resolution rate is an output. It's the sum of every small correctness question in the journey, and each one of those is testable. If you can't say which specific step your self-service is losing people at, you don't have a self-service adoption problem yet. You have a measurement problem that is producing one.

The workforce cost of untested automation

The third finding deserves a connection that the interview doesn't make.

When automation fails quietly, the failure doesn't vanish. It lands on a human. The caller who couldn't get the bot to understand them arrives at an agent already annoyed, having repeated themselves four times, and the agent starts the conversation forty seconds into a deficit that wasn't their fault. Do that a few hundred times a day and you've built a working environment that grinds people down.

So the workforce risk and the testing gap are the same risk seen from different chairs. Kyle's point is that badly managed AI transition costs you good staff. Part of managing it well is not shipping automation that offloads its own defects onto the people you're trying to keep.

So why isn't testing on the list?

Because for most of the last two decades, testing your voice channel properly was not economically available to you.

The incumbents are good at what they do. Cyara has the broadest portfolio in this space and the longest track record in regulated industries, and Hammer has DTMF and SS7 depth nobody has replicated. I'd genuinely recommend either in the right situation, and I've written that out honestly elsewhere. But the cost of that breadth is enterprise procurement. Custom quotes, professional services, multi-year commitments, and a per-journey cost high enough that covering everything was never on the table.

So teams did the rational thing under that constraint. They picked the five or ten journeys that mattered most, automated those, and handled the rest the way everyone handles the rest.

Someone from the ops team calls the number on release morning.

That's the actual state of the art in a lot of very serious organisations. A human being with a mobile phone and a list, working through as much as they can before the change window closes. It is well-intentioned, it is done by people who care, and it is a sampling strategy with a sample size of one. It tests the happy path, at one time of day, from one network, with one accent, by someone who already knows what the bot is supposed to say and will unconsciously help it along.

It misses the things that break. Specifically, it misses the sub-menu nobody thought to check, the journey that only fails on the second attempt, the flow that works fine at 11am and falls over when 400 people call at once, and the regression that reappeared in a component nobody touched. Those become outages, and outages on a phone line are invisible until they're loud. There's no error page to screenshot. Just callers who couldn't get through and mostly didn't tell you.

In a regulated industry it's worse than an outage. The manual test on release morning was never going to check whether the bot disclosed it was an AI, or whether it accepted "uh-huh, sure" as consent for a policy renewal. We pointed an audit at one live UK number and ten of eighteen tests failed, six of them critical. Nobody at that organisation was being careless. They just had no mechanism that would ever have told them.

What full coverage looks like when it's affordable

The reason I think testing belongs on the up-at-night list now is that the constraint that kept it off has gone. Real phone calls, driven by synthetic customers, stopped being expensive. Which means partial coverage is now a choice rather than a budget consequence.

In practice, covering an estate properly needs four different shapes of test, and it's worth being concrete about them.

A single test, for one journey you care about. In TotalPath that's one of three kinds. A Standard test scripts the call step by step, waiting for expected phrases, pressing digits, checking transcript similarity against a threshold you set. A Parameterised test is the same thing as a reusable template with variables, so one definition covers many data combinations. A Mission test hands an AI caller a goal and an acceptance criterion and lets it hold a real conversation, then judges the outcome against the criterion. That last one is the only shape that works properly against an agentic bot, because there's no fixed script to assert on.

A batch, for regression. Group your test configs into a batch and they all run in parallel on every fire. You get one aggregated result with passed, failed and errored counts, and you can re-run just the failures without redoing the whole set. This is the bit that replaces release morning. It isn't ten journeys because ten was the affordable number, it's every journey you've ever written a test for, and it finishes in the time the slowest single call takes.

A schedule, because your bot changes when you aren't looking. Attach a schedule to a test or to a whole batch, either a recurring interval or a cron expression in your timezone, with alerts on failure. This is the part teams skip and shouldn't. A voice channel drifts continuously. Models get updated, prompts get tweaked, a vendor ships something on a Tuesday. Point-in-time QA is a claim about a version that no longer answers your phone. If you want a longer argument for this, it's the whole reason we flip the testing pyramid.

A load test, because concurrency breaks things correctness testing never will. Shape a load curve, set your duration and peak concurrency, and run your existing test config under it. What comes back is a success rate plus P50, P95 and P99 response times, broken out across ramp-up, steady state and ramp-down. Almost every "the IVR was fine in testing" incident I've seen was a concurrency failure. One call at a time proves nothing about four hundred, and most teams load test agentic IVRs wrong when they do it at all.

Every run leaves the same evidence behind. The call recording, a word-level transcript, per-step expected versus actual with a similarity score, turn latencies, dead air, and for mission tests a verdict that quotes the line from the transcript it was based on. That matters because it converts an argument into a fact. Nobody has to agree about whose fault it was. The transcript settles it.

The fourth thing

Kyle's three findings are the right three, and the research behind them is better than most of what gets said at a conference. Fragmented analytics really is the defining operational problem. Self-service really is failing to keep up with the demand for it. The workforce transition really is being deprioritised, and that really will cost more than it saves.

I'd just note that all three get harder to fix while you can't answer a much smaller question. Does the journey work? Not "did NPS move" or "what does the dashboard suggest" or "did the vendor demo well." Does a customer who calls your number right now, with an ordinary need, get it done.

You can build the analytics platform and still not know. You can redesign self-service and still not know. You can manage the workforce transition beautifully and still be handing your agents the fallout from automation nobody checked.

That question has been answerable for a while now. It just never made the list, because for years the honest answer was "we'd love to know, and we can't afford to." That stopped being true, and it's worth updating the list.

Frequently asked questions

Isn't this just monitoring under a different name? Monitoring watches real traffic and tells you when real customers are already affected. Testing runs a customer you invented, so the first thing through a broken journey is synthetic. Both are useful. Only one of them is preventative.

We have contact centre analytics already. Why would we test? Analytics tells you what happened. Testing tells you what will happen. If your analytics is fragmented across systems, as Kyle's research suggests it usually is, a test also gives you the one thing the fragments can't, which is a single continuous record of one complete journey.

Our release-morning manual test has never missed anything. It's missed things. You just found out about them as complaints, repeat calls, or abandoned sessions rather than as a failed test, so they never got attributed back to the gap that caused them.

How many journeys should we cover? All of them, now that you can. The useful question changed from "which journeys can we afford to test" to "which journeys are we content to leave untested," and that second question is much harder to answer in a meeting.

Does this work on our stack? Tests run as real calls over the PSTN, so anything terminating a phone number is testable regardless of what sits behind it. No integration required.

---

If the three things on the list are keeping you up, the fourth one is probably the cheapest of them to fix.

Talk to us

Or register for a free account and run one against your own number before the next release.