If your contact centre AI learns, who checks what it learned?

Claudio Rodrigues, chief product officer at Omilia, gave an interview to CX Today last month that is worth your twenty-five minutes. It's on YouTube in full. The headline - "Your Contact Center AI is Failing – And You Probably Built it That Way" - is considerably more combative than anything he actually says. What he actually says is careful, and mostly right.

I want to pick up his last argument, because I think it contains a problem he doesn't quite finish. He is arguing for a contact centre AI that keeps learning after go-live. I agree. But almost nobody has noticed what that does to testing.

TL;DR

• Rodrigues is right that a static contact centre AI depreciates, and that a dynamic one is the thing worth buying.
• A dynamic asset is precisely the thing you cannot test once. Every voice QA process in the industry is triggered by a deploy. A self-learning system has no deploy.
• He's also right to move from containment to resolution. But resolution measured from the platform's own logs is the system marking its own homework.
• If the system learns from its own behaviour and grades its own behaviour, the loop has no external reference point. Something outside it has to call the line and form its own opinion.

What he gets right

Start with the thing that will annoy people the most, because it's the most correct.

Rodrigues says AI deployments fail long before the technology gets a chance to: "it starts from a poor definition of the problem." Teams decided they needed agentic AI, then went looking for a problem it could be pointed at. He is blunt that the answer sometimes isn't AI at all - that a heuristic would do the job better, and that this is fine. That's an unusually honest thing for a vendor of AI to say out loud, and it matches what we see. The failures we get called into are rarely mysterious. Somebody automated a journey nobody had defined, against a metric nobody had agreed.

His second point is that problem definition and metrics are the same conversation. If you know the problem, the metric falls out of it. If you can't name the metric, you didn't have a problem, you had a budget line.

Then the good bit. Asked about the difference between an AI built to handle queries and one built to learn from them, he doesn't hedge:

One is a static asset, the other is a dynamic asset.

And: "nobody wants to do an investment on something that depreciates in value." A system frozen at its go-live date is answering last year's questions. Your customers moved. Containment quietly slides, and the fix is a rebuild.

This is right, and it's the direction the whole category is moving. It also has a consequence he doesn't chase.

A dynamic asset can't be tested once

Every voice testing process I have ever seen - ours included, and certainly the incumbents' - is triggered by change. Somebody deploys. A prompt is edited, a flow is republished, a model is swapped, a recogniser is upgraded. That event fires the regression suite. Test, compare to baseline, ship or roll back.

That model has a load-bearing assumption underneath it: the system under test holds still unless someone changes it.

A self-learning system violates that assumption by design. That's the whole product. It ingests real interactions and adapts. Which means the thing answering your phone on Friday is not the thing you tested on Monday, and nobody deployed anything. There's no commit. No release note. No diff to review. No change ticket. Nothing to trigger the suite.

So the trigger has to stop being change and start being time. Not because you're paranoid, but because arithmetic: if the asset is dynamic, "we tested it at launch" is a statement about a system that no longer exists.

I wrote before about flipping the testing pyramid for contact centres. This is the sharper version of the same point. A static IVR justified a fixed suite run at release. A self-learning agent needs continuous assurance on a schedule, because its release cadence is now continuous and invisible.

Resolution is the right metric. Measured by whom?

The strongest section of the interview is on metrics, and I'd take it one step further than he does.

Rodrigues doesn't rubbish containment - he's careful, he calls it the right metric that no longer captures the whole picture. He prefers resolution, and his reasoning is about language: "resolution feels like we solved the problem for the customer," where containment always carried a faint whiff of deflecting calls away from an agent. He's right. It's a better word because it describes a better goal.

Here's my problem. Resolution is much harder to measure than containment, and the difference matters.

Containment is a fact about your own infrastructure. Did the call reach a human? Your platform knows. It cannot really lie to you.

Resolution is a claim about the customer's life. Did they get the thing they rang for? Your platform does not know that. It knows the flow reached the node labelled success. That is not the same fact, and the gap between the two is where every unhappy customer lives.

So when a self-learning platform reports its own resolution rate, from its own logs, using its own model to judge whether the conversation went well - that's the system marking its own homework. Rodrigues talks about the value of "listening to every customer call and understanding what is it that they are actually talking about." That's genuinely valuable. But it's the AI observing itself, scored by the same class of model that produced the behaviour, and inheriting the same blind spots.

Notice that he raises the failure mode himself. He brings up the industry worry about "the implosion of AI quality based on AI just training on AI data," and answers it by saying humans will always guide the system. But look at the guidance he describes: strategy, brand pillars, guidelines, what good looks like. That's a manager setting direction - which is correct, and which I'd take over an annotation sweatshop any day.

It just isn't a measurement. Direction isn't a control. A loop where the system acts, learns from its own actions, and grades its own results is open at exactly the point where you need it closed.

What an outside observer is for

The only way to know whether your voice AI resolves anything is to have something that isn't your voice AI ring it up and try.

That's what TotalPath does, and it's why the model is what it is. A synthetic customer calls your real number over the real PSTN, pursues an actual objective - reschedule the delivery, dispute the charge, get a human - and is graded against acceptance criteria written by you, not inferred by the system under test. The verdict comes from outside the platform's own telemetry.

Three things follow from that, and they map onto exactly the gaps in Rodrigues' argument.

The tests assert on outcome, not on strings. We assumed non-determinism from day one because agentic bots don't repeat themselves. A mission test cares whether the objective was achieved, not whether a phrase was uttered. That's the only assertion model that survives a system that rewords itself weekly.

The schedule is the trigger. Because a self-learning system has no deploy event, we run on a cadence and diff against a baseline. Drift shows up as a trend, not an incident. You want to see resolution sagging over three weeks, not discover it in a QBR.

The observer is independent, on purpose. Our grade doesn't come from your logs. It comes from a call we placed and a transcript we assessed against your criteria. When your platform's dashboard and our report disagree, that disagreement is the most valuable thing you'll read all month.

And to be fair to Omilia: none of this is a knock on them specifically. A self-learning platform that reports its own resolution rate is doing nothing dishonest. It is doing the only thing it can do from the inside. The point is that inside is not where this measurement can be taken from.

What good actually looks like

Rodrigues was asked what CX leaders should expect and aren't getting. He said faster go-live, better metrics, and a line of sight from AI to growth. Fair.

I'd add a fourth, and I think it's the one that makes the other three safe to want:

Evidence, from outside the system, that the thing still works this week.

Not a dashboard the platform generates about itself. Not a containment number that was true in March. A real call, placed at a real time, from a real network, graded against criteria you wrote, with a record you can hand to an auditor when they ask what your bot was saying to vulnerable customers in Q2.

Buy the dynamic asset. He's right about that. Just don't accept the dynamic asset's own account of how it's doing.

Frequently asked questions

Isn't this just monitoring? No. Monitoring tells you the system is up and the function call ran. This tells you a customer could achieve something. Those diverge constantly - a perfectly healthy platform can fail every caller.

Can't the platform test itself on a schedule? It can, and several do, and it's better than nothing. But it's the same model judging the same behaviour on the same data. Independence is the whole point.

Does this apply to non-self-learning bots? Yes, but less urgently. A static bot at least changes only when you change it, so a deploy-triggered suite catches most of it. Self-learning removes that safety net.

We're on Omilia / Genesys / Amazon Connect / Vapi. Does that matter? No. We test over real PSTN, so anything that terminates a phone number is testable. We're vendor-neutral on purpose - that's what makes us a usable second opinion.

---

If you've deployed something that learns, and you'd like an outside line on whether it's still doing what you hired it for, that's the job we built TotalPath to do.

Talk to us

Or register for a free account and point a call at your own number in the next five minutes.