Third-party voice AI isn't ready for regulated environments, and the missing piece isn't the model

There's a thread on r/VoiceAutomationAI this week that describes a pattern anyone selling into banks or healthcare will recognise immediately. The agent works. Latency is fine. The demo goes great. Then it reaches security review and stalls for weeks, and the questions that stall it are always the same four: how do you stop it disclosing account details to the wrong caller, how do you verify who is on the line before it takes an action, what happens when a caller talks it out of its instructions, and who gets alerted when it goes off script mid-call.

The teams in that thread answered with prompt rules and post-call transcript review. Reviewers rejected both. One reviewer's summary, quoted in the thread, is the most efficient description of the problem I've read: so you've asked the AI nicely not to do the bad thing.

I want to argue that the reviewers are right, that this is not solvable by better prompting or a better model, and that what's actually missing is a piece of architecture almost nobody has built yet.

TL;DR

• Testing has largely caught up with the conversation. We can measure whether a voice agent understands, responds in time, survives concurrency, holds up under adversarial pressure, and says the things the law requires it to say.
• Nothing has caught up with the transaction. Between "the model decided to do a thing" and "the thing happened to a customer's account" there is, in most deployments, no enforceable boundary.
• Prompt injection has no general fix. OWASP has kept it at number one since the list began, and practitioners are increasingly blunt that it remains unsolved at the architectural level because models process system instructions and untrusted input as one token sequence.
• So the control cannot live in the model. It has to live in code the model calls into, and that code has to be yours, because it encodes your identity policy, your entitlements, your retention rules and your regulator.
• That layer has a shape, and the shape is now written down in several places. It just isn't in the voice stack yet.

What we actually can test now

I'll concede the optimistic half of the argument first, because it's real and it's the part of the market I work in.

Voice quality is measurable. Intent recognition against accents, background noise and disfluency is measurable. Turn latency, dead air and barge-in behaviour are measurable. Concurrency is measurable, and most teams get that one wrong when they attempt it at all. Adversarial robustness is increasingly measurable too: you can now hand an AI caller the goal talk this agent out of its instructions and judge what came back, which is a genuinely new capability and one that didn't exist two years ago.

Regulatory conversation content is measurable. We pointed a compliance pack at one live UK number and ten of eighteen tests failed, six of them critical — things like failing to disclose it was an AI, or accepting an ambiguous "uh-huh" as consent. That matters more this month than it did last month, because EU AI Act Article 50 became applicable on 2 August 2026, and it requires that people are informed they are interacting with an AI system, and that an AI agent discloses the identity of the person or entity it acts on behalf of. That is a testable property of a conversation, and it is now a legal one.

So: the conversation layer is in decent shape. If your objection to voice AI is "we can't prove it behaves properly on a call," that objection is weaker every quarter.

The part nobody can evidence

Now the other half.

Consider the example the Cartesia engineer works through in that Reddit thread — a caller says "I've moved, can you update where my statements are sent?" It's a mundane request. It's also an authenticated write to a system of record, initiated by a voice on a phone line, mediated by a probabilistic model.

Here is what has to be true before that write is safe, and here is why each part is currently hard.

Identity. The caller has to be authenticated before anything sensitive happens, and caller ID and voice similarity cannot be the thing that does it. FinCEN's November 2024 alert on generative-AI fraud documents fraudsters answering liveness checks with synthetic audio; voice as a credential is a credential your attacker can now manufacture. Meanwhile the fallback most contact centres reach for is knowledge-based authentication, and NIST SP 800-63B now states plainly that verifiers "SHALL NOT prompt subscribers to use knowledge-based authentication (KBA) ... or security questions." Its authenticator catalogue has no knowledge-based option in it at all. So the two mechanisms voice channels have historically leaned on are, in 2026, both off the table as primary controls.

Authority. Even a correctly authenticated caller is only allowed to do certain things to certain accounts. That decision is an access-control decision, and access-control decisions belong in code with a policy engine behind them, not in a system prompt. OWASP's LLM06, Excessive Agency — the entry that expanded most in the 2025 revision — is precisely the failure mode where an LLM gets more functionality, permission or autonomy than the task requires, and can therefore be steered into actions nobody intended.

Credentials. This is the one I think is most underappreciated, and it's the reason I say the missing thing is an integration layer rather than a policy. When the agent calls your CRM, whose token does it present? If the answer is "a service token belonging to the agent platform," then your CRM's audit log records the platform, not the customer, and any successful manipulation of the model inherits the platform's full authority. This is the classic confused deputy, and it is now written down as a normative rule in the MCP security specification: servers "MUST NOT accept any tokens that were not explicitly issued for" them, with token passthrough named as an explicitly forbidden anti-pattern for exactly the reasons a bank reviewer would give — bypassed rate limits, broken audit trails, and a trust boundary that lets one compromised service reach every connected one.

Retention. Whatever the caller says gets recorded and transcribed by default. If any of it is card data, PCI DSS requirement 3.3.1 prohibits retaining sensitive authentication data after authorisation even if encrypted, and the Council is explicit that this includes card validation values captured in digital audio recordings. Add a third-party ASR vendor, an LLM provider, a TTS provider and an analytics pipeline, and you have four copies of that audio in four jurisdictions before anyone has thought about redaction.

The supply chain itself. For EU financial entities this is no longer a matter of taste. DORA has applied since 17 January 2025, and Article 28 requires a maintained register of every ICT third-party contractual arrangement — including the subcontracting chains underneath it — plus a pre-contractual assessment covering substitutability, insolvency risk and data protection. A modern voice agent is typically five vendors in a trench coat. Try filling in the register for one.

The layer that's missing

Notice that not one of those five problems is a model problem. Every one of them is a problem about the boundary between the model and everything it can touch, and boundaries are the thing our industry has been vague about.

The shape of the fix is not mysterious. It looks like this, and it is worth stating as an explicit division of labour: the model proposes, code disposes.

The model does what it is genuinely good at — interpreting messy human speech, holding a natural conversation, mapping "I've moved" onto an intent. It does not get to decide whether the intent is permitted, who the caller is, or whether the write happens. Between the model and your systems sits a policy enforcement point that:

• exposes a narrow, explicit catalogue of operations rather than general API access, and rejects anything the model proposes that isn't on the list for this agent;
• authenticates out of band — a one-time code to a registered channel, an authenticated app flow, something that isn't the audio stream itself — and treats caller ID and voice match as signals, never as grants;
• authorises against the authenticated identity, not against the agent's identity, using tokens minted for that caller and that operation, so your downstream audit log names a customer rather than a vendor;
• validates arguments against a typed schema before anything executes, so a hallucinated postcode fails as a type error rather than becoming an address change;
• creates a pending action and binds confirmation to it specifically, so that an interruption, a topic change or an unrelated "yes, sure" cannot authorise a transaction that was never read back;
• re-checks policy at execution time — authentication freshness, entitlement, rate limit, approval requirement — because the conversation may have moved on since the check that happened forty seconds ago;
• redacts at capture rather than after, because the PCI obligation attaches to the recording, not to the report;
• and writes a decision log separate from the transcript: operation, policy verdict, confirmation state, tool result, escalation. That log is the artefact a security reviewer actually wants, and it is the one thing a transcript can never be.

Read that back and you'll notice it's a state machine with an authorisation model bolted to it, and that it has almost nothing to do with speech. That's the point. It's plumbing, it's unglamorous, and it is the entire difference between a demo and a deployment.

Why your voice platform can't own it for you

The obvious follow-up in the thread was: fine, so will the platform build this? Cartesia's own answer was that they provide managed agents and the speech models, but the hooks into enterprise systems have to be owned by the implementing team. Another commenter put the structural reason more sharply: authentication and entitlement are deeply bound to your company's data and policy, so a general-purpose voice vendor can't own them without becoming a bespoke integration shop for every customer.

I think that's correct, and there's a second reason underneath it. These platforms are fast because their pipelines are tightly optimised end to end. Injecting your identity provider, your policy engine and your core banking latency into the middle of that pipeline degrades the exact thing you were paying for. You either accept a slower agent or you accept a thinner boundary, and most teams under launch pressure quietly accept the thinner boundary and hope the transcript review catches it.

It doesn't catch it. Transcript review is a detective control operating after the money moved, and no bank's second line has ever accepted a detective control as a substitute for a preventive one.

This is also why "who owns the voice agent" is such a persistent organisational mess. The conversation belongs to CX, the model belongs to the AI team, the systems of record belong to core engineering, and the layer I've just described sits in the gap between all three, which is exactly why nobody has funded it.

Where testing fits, honestly

I'm not going to pretend that testing solves this, because it doesn't. You cannot test your way to a control that does not exist. If your agent authenticates by asking for a date of birth and then calls your CRM with a platform service token, there is no test suite in the world that makes that safe.

What testing does is convert the layer from a claim into evidence, and that is the specific thing the security reviews in that thread were failing on. Once the boundary exists, it becomes testable in the ways reviewers accept:

• Adversarial mission tests — hand an AI caller the goal of extracting another customer's details, or of talking the agent into skipping a verification step, and judge the outcome against an acceptance criterion. This is red-teaming as a scheduled regression suite rather than a one-off exercise.
• Confirmation-binding tests — interrupt mid-readback, change topic, then say "yes." A correct implementation must not treat that as authorisation. This is a scripted test, and it fails a surprising number of agents.
• Authentication bypass tests — attempt the sensitive operation before the out-of-band step completes, and after the authentication window should have expired.
• Disclosure tests — Article 50 compliance is a phrase-level assertion on a real call, which is about as testable as things get.
• Concurrency — because authorisation checks that pass serially routinely fall over at four hundred simultaneous calls, and a policy engine timing out under load usually fails open unless someone specifically decided otherwise.

Every one of those leaves the same artefact behind: a recording, a word-level transcript, per-step expected versus actual, and a verdict that quotes the line it was based on. That's what turns "we've thought about this" into something a reviewer can sign. And because a voice agent changes underneath you whether or not you touch it, it has to run on a schedule rather than once at launch — a point-in-time assurance is a claim about a version that no longer answers your phone.

So what should you actually do

If you are a regulated buyer, the useful question to put to a voice AI vendor is not "is it accurate" or "what's your latency." It's: show me where in your architecture an authenticated identity is bound to an authorised operation, and show me the log that proves it. If the answer involves the words "system prompt," you have your answer.

If you are building one, build the boundary first and the personality second. The boundary is the part that takes six months and the part that gets you through review.

And if you are a platform vendor, the honest position — the one Cartesia took in that thread, to their credit — is that you provide the speech and the orchestration and the customer provides the trust boundary. That's a defensible split. It's just one the market's marketing has been reluctant to say out loud, and every buyer discovers it in week three of security review instead.

Voice AI is not unready for regulated environments because the models aren't good enough. They're good enough. It's unready because the integration pattern between a probabilistic conversation and a deterministic system of record is still being invented, in public, one stalled security review at a time.

Frequently asked questions

Isn't this just guardrails? No. Guardrails usually means filtering or classifying model output, which is still a probabilistic control on a probabilistic system. What I'm describing is a deterministic authorisation boundary that the model cannot talk its way past, because it isn't asked.

Can't a good enough model be trusted with this? Not on current architecture. The reason prompt injection persists is that models process instructions and untrusted input in the same token stream, with no privilege boundary between them. A better model raises the cost of an attack; it doesn't create the boundary.

We're not in the EU or the US. Does any of this apply? The regulations differ, the failure modes don't. Disclosing account details to the wrong caller is a bad day in every jurisdiction. The standards cited here are useful as a checklist even where they aren't binding on you.

Our vendor has SOC 2 and ISO 27001. Those attest to how the vendor runs its own controls. Neither says anything about whether a caller can talk your agent into changing someone else's address, which is an application-design question about your deployment.

Where does this leave the pilot we're running? Probably fine, if it's read-only. The line worth drawing is between agents that inform and agents that act. Informational agents have a fraction of this problem. The moment a call can write to a system of record, everything above applies.

---

If you're heading into a security review with a voice agent, the fastest thing you can do is find out what it currently does when someone tries to talk it out of its instructions.

Talk to us

Or register for a free account and run one against your own number before the reviewer does.