How to test an AI agent before customers use it

A demo conversation that goes well proves very little. The questions that matter are what the agent does with the odd, rude, urgent or ambiguous message, and whether a person finds out when it should. This guide gives you a test plan you can run in a few days, without needing to read code.

Before you test: write down what "correct" means

You cannot test against a feeling. Agree three short lists first:

• Must do: the outcomes every conversation should reach, such as collecting the address or offering real booking times.
• Must never do: quote prices, promise dates it cannot check, give medical or legal advice, reveal other customers' details, argue with a complaint.
• Must hand over: the triggers that send a conversation to a person, with who receives it and how fast.

The acceptance-test checklist

Normal conversations

• Each service you offer, asked about in plain words and in shorthand.
• Each step of booking: offer, choose, confirm, reschedule, cancel.
• Questions answered from your approved information: hours, area, what to expect.
• Spelling mistakes, voice-to-text errors and one-word replies.

Edges and limits

• A request for something you do not offer, or an area you do not cover.
• A price question, phrased several ways.
• A customer who changes their mind halfway through.
• A message in another language, if your customers might send one.
• A long, rambling message with the important detail at the end.

Handover

• Every handover trigger fires, and the right person is notified with a usable summary.
• After a person replies, the agent stays quiet on that conversation.
• Out-of-hours handovers say honestly when someone will respond.

Safety and misuse

• Attempts to override its instructions ("ignore your rules and…"). OWASP lists prompt injection as the first risk for applications built on language models.
• Requests for personal or internal information it should not share, which OWASP calls sensitive information disclosure.
• Requests that push it to act beyond its job, such as cancelling another customer's booking. Limit what it is able to do, not only what it is told to do; OWASP calls this excessive agency.
• Opt-out words such as STOP, and requests to delete their details.

Failures

• Calendar or CRM unavailable: the agent does not claim success, and a person is alerted.
• Same customer messages on two channels at once.
• Very slow replies from the customer, over hours or days.

Sample test records

Build a small set of test messages and the behaviour you expect from each. The examples below are invented test records, labelled as such. Use names and numbers you create for testing, never real customer data.

How to run the tests

1. Use a test number or test contact that is clearly labelled, so nothing reaches a real customer or your live reports.
2. Run every test at least three times. Language models do not answer identically each time, so one pass is not enough.
3. Record pass or fail for each run against the expected behaviour, with a note on anything surprising.
4. Have someone who did not write the rules run a second round. They will phrase things differently.
5. Fix, then rerun the whole set, not only the test that failed. A fix in one place can change behaviour elsewhere.

Sign-off before customers see it

• Every "must never" test passes on every run.
• Every handover trigger reaches a named person, and they confirmed they received it.
• The owner has read at least twenty full test conversations.
• There is a written way to switch the agent off quickly, and someone knows how.
• A review routine is agreed for the first month: who reads live conversations, how often, and how fixes are requested.

After launch

Testing does not end at launch. Real customers will find phrasings you did not think of. Read a sample of live conversations every week for the first month, and add each new surprise to your test set so it is checked every time the rules change. If you want a broader structure, NIST's voluntary AI Risk Management Framework covers managing AI risk across design, use and evaluation.

Limits

• Passing these tests reduces risk; it does not remove it. Keep a person responsible for the workflow.
• Voice agents need extra tests for accents, background noise and interruptions, which this checklist only touches on.
• Regulated industries have their own requirements for records and advice. Follow them in addition to this list.