How mission testing works

Standard or mission?

There are two ways to test a phone line, and they answer different questions.

• A standard test follows a script you write. Each step listens for words you expect, presses a key or plays audio. It passes when every step matches. Use it when you know exactly what the line should say: a menu, a recorded message, a press-1-for flow.
• A mission test gives an AI caller a goal. You say what the caller is trying to do and what counts as success. The AI caller rings the line, holds a natural conversation, and afterwards the call is judged against your success criterion. Use it for voice agents and conversational flows, where the wording changes from call to call but the outcome shouldn't.

Both are created from Testing → Create Test. Saved standard tests are listed under Test Config and their results under Standard Results. Mission tests have their own pages: Mission Test Config and Mission Results.

The building blocks

A mission test draws on things you set up once and reuse across many tests.

1. Data (Data Management → Data). Datasets hold the caller details your tests use: name, date of birth, ZIP code, phone number, email, account numbers. Each item has a key and a value. Mark PINs and passwords as Secret so they are masked on screen.
2. Personas (Data Management → Personas). A persona is who is making the call. It picks the items it needs from your datasets; it points at them rather than copying them, so changing a value in a dataset changes it for every persona that uses it. The test wizard calls this field Data Profile. They are the same thing.
3. The mission test (Testing → Create Test → Mission). The number to call, the mission, the success criterion, the persona, and the caller's voice and language.
4. Runs (Mission Results). Every time you run the test it places a real call and saves the transcript, the recording and the verdict.

Build in that order: data, then personas, then tests. A mission test can't be saved without a persona, and a persona is only as useful as the details it carries. If the line asks for a date of birth and the persona doesn't have one, the call stalls there.

One persona, many tests. For a set of tests against the same line, build one persona per kind of caller, for example an adult patient and a parent booking for a child, and reuse it. Name personas for the job they do in your tests rather than after a real person.

Writing the mission

The mission is the AI caller's brief. Write it to the caller, in plain language, as you would brief a colleague making the call for you. Say what they want, and what to do at the points where the line could go more than one way.

You are calling to book a flu shot for yourself. They will ask for your name, date of birth, ZIP code and which store you'd like. Accept the first appointment they offer on a weekday. If they offer to send a confirmation, ask for it by text. If they say the last four digits of your phone number are on file and ask whether to use that number, tell them to use 415 320 9561 instead.

• Give it the decisions, not a script. "Accept the first weekday slot" works better than the exact words to say, because the line won't ask in the same words every time.
• Put the caller's details in the persona, not the mission. The AI caller already has the persona and gives those details when asked. It reads numbers digit by digit and spells names and email addresses letter by letter when asked to. Use the mission only for things that differ from the persona, like the phone number in the example.
• Say what to do when things go wrong, if that matters to the test, for example "if they can't find a slot, ask for a pharmacist". Without that, the AI caller acts like a reasonable caller: when the line goes round in circles, it asks for a person, and hangs up if that doesn't help.
• Set the maximum call length to fit the flow. A booking that collects a name, date of birth, address and email often runs past five minutes. The call is ended when it reaches the limit, and a call cut short usually fails.

Writing the success criterion

The success criterion (shown as Acceptance criterion) is what has to have happened for the test to pass. After the call, the transcript is judged against this sentence alone, so write it the way you would mark the call yourself.

The mission is a success if a flu shot appointment is booked and the agent confirms the date, time and store. The phone number 415 320 9561 must be the one used for the appointment. Being transferred to a person before the booking is confirmed is a failure.

• Make it observable. "The appointment is confirmed with a date and time" can be checked in a transcript. "The caller is helped" can't, and it gives different verdicts from run to run.
• Name the failures that look like success, such as being transferred instead of served, or an answer in the wrong language.
• Test one outcome per test. If you want to check both the booking and the confirmation text, either write both into the criterion or make two tests, so a failure tells you which part broke.

Testing in another language

Two settings on the persona step decide how a call sounds and what it can understand.

Call language is the language the line will speak, so the call can be understood. Pick it to match how the line behaves:

• English (the default) for an English line.
• A single language, such as Spanish, when the line speaks that language from the first word. The AI caller then speaks only that language, numbers, dates and spellings included.
• More than one language when the line answers in one language and switches. Use it when the line greets callers in English and the caller asks for Spanish, or chooses it from a menu. It understands both in the same call.

With More than one language, say in the mission when to switch and how. A real caller can ask for their language, or press the key the menu offers:

As soon as the call is answered, ask in Spanish for the call to continue in Spanish, and speak only Spanish from then on.

If the menu offers Spanish (for example "para español, oprima nueve"), press 9.

If the call language doesn't match the line, the AI caller can't understand what it hears. A Spanish answer heard as English comes out as nonsense in the transcript, and the call goes wrong from there. Check this setting first when a transcript doesn't make sense.

Voice is what the AI caller sounds like. The list shows each voice's language, accent and gender. Search it by language, for example "Spanish", and press play to hear a voice before you choose. Pick a voice that speaks the call's language. An English voice speaking Spanish is understood, but it sounds like a learner, and some systems handle that less well.

Write the success criterion with the language in mind. If the line must stay in Spanish once asked, say so; otherwise a call that drifts back to English can still pass.

Reading the result

Each run on Mission Results shows:

• The verdict, pass or fail, with the reasoning and the lines from the call it rests on.
• The transcript, with the line's words and the AI caller's turn by turn, and the recording of the whole call. The recording can take a minute or two to appear after the call ends.
• Call analyses such as silence gaps and response latencies, for how quickly and smoothly the line responded.

When a run fails, read the reasoning first, then the transcript around the lines it quotes. The usual causes:

• The line didn't do what it should. This is the failure the test is there to find.
• The persona was missing a detail the line asked for, so the call stalled.
• The call language didn't match the line: look for garbled lines in the transcript.
• The criterion asked for more than the mission did, such as a confirmation text the caller was never told to ask for.
• The call hit the maximum length before the flow finished.

When you ask us about a run, include its run ID so we can find the call.

Next steps

• To run a set of tests together, or on a schedule, see Batches and Scheduling.
• To create and run mission tests from your own code, see Mission Test Configs and Mission Tests.