How to Test an AI Chatbot Before Launch: Evals, Test Sets and Release Gates for Australian Businesses (2026)

Somewhere in the first week after most chatbot launches, someone on the team screenshots an answer the bot should never have given. A refund policy that doesn’t exist. A delivery promise nobody can keep. A confident description of a product you stopped selling in March. Nobody tested for it because nobody knew it was possible, and the demo went so well.

That gap between “it answered my five questions nicely” and “it behaves properly across the thousand questions real customers will ask” is where most AI projects quietly go wrong. It isn’t a model problem. It’s a testing problem, and it’s one that traditional software teams solved decades ago with test suites. AI features need the same discipline, adapted for the fact that the output is different every time you ask.

This guide covers how we test AI chatbots, internal assistants and AI-powered features before they go live and while they run: building a test set from your own data, scoring answers without fooling yourself, using another model as a marker (and where that goes wrong), and keeping the whole thing cheap enough that a small team will actually keep doing it. If you’re still choosing what to build, our guides on AI chatbots for customer support and internal AI assistants cover that side. This one starts at the point where you have something that works in a demo and need to know whether it works in real life.

Why “it seemed fine when I tried it” isn’t testing

When someone builds an assistant, they test it the way they’d test a form: type in a few inputs, check the outputs look right, move on. That works for deterministic software because the same input always produces the same output, so five good results tell you something about the sixth.

Language models don’t behave that way. The same question asked twice can produce two differently worded answers, and occasionally two different answers. A small change to the prompt, the documents behind it or the model version can fix the five questions you were looking at and quietly break fifty you weren’t. Five good results tell you almost nothing about the next hundred.

There’s also a human problem. The person who built the bot knows how to ask it questions it can answer. They phrase things the way the documents phrase them. Customers don’t. They write “do u deliver to WA”, they paste an order number with a typo, they ask about a product by the nickname the warehouse uses, and they ask three things in one message. The builder’s testing is biased toward success without anyone intending it.

The cost of getting it wrong is real, and it lands on you

It’s tempting to treat a wrong chatbot answer as a harmless glitch. It isn’t always. In the 2024 Moffatt v Air Canada decision, a Canadian tribunal held the airline responsible for incorrect bereavement-fare information its website chatbot gave a customer, and rejected the argument that the chatbot was a separate entity responsible for its own statements. That was a Canadian case and not binding here, but the reasoning will sound familiar to anyone who knows Australian Consumer Law: under the misleading or deceptive conduct provisions, what your business says to customers is your statement, regardless of whether a person, a template or a model wrote it.

We’re not lawyers and this isn’t legal advice. The practical point is simple. If your assistant states a price, a policy, a warranty term or a delivery time, treat those statements as published commitments and test them the way you’d check a printed brochure.

What an eval actually is

“Evals” is the term the AI world uses for automated tests of model behaviour. Strip away the jargon and an eval is three things: a question, a description of what a good answer looks like, and a way of checking the real answer against that description. Run a few dozen to a few hundred of those every time something changes, look at what got worse, and you have a regression suite.

The important mental shift is that you’re not testing for one exact output. You’re testing for properties. “Mentions the 30-day return window” is a property. “Doesn’t promise a refund the policy doesn’t allow” is a property. “Offers to hand over to a person when the customer says they’re angry” is a property. A wide range of wordings can satisfy each one, which is exactly what makes them workable for systems that never answer the same way twice.

The anatomy of one test case

A usable test case has four fields, and you can keep them in a spreadsheet:

• Input: the message exactly as a customer might send it, typos and all.
• Context: anything the bot would know at that moment, such as the customer’s logged-in state, the page they’re on, or the order being discussed.
• Must include / must not include: the facts or behaviours a pass requires, and the ones that cause an automatic fail.
• Category and severity: what kind of test it is and how bad a failure would be, so a wrong opening-hours answer doesn’t weigh the same as a wrong refund promise.

That’s the whole format. Teams get tangled in tooling before they’ve written a single case, and the cases are the valuable part. A spreadsheet of sixty honest, awkward, realistic cases beats an elaborate platform with ten tidy ones.

Building a test set from what you already have

The best source of test questions is what your customers and staff already ask. You almost certainly have it sitting in inboxes, helpdesk tickets, live chat transcripts, call notes and the “contact us” form. Inventing questions from scratch produces the polite, well-formed ones. Real history gives you the messy ones.

Where to mine questions

Start with the last three to six months of support tickets or emails. Pull a few hundred, strip names and personal details before anyone else sees them, and sort them by topic. You’re looking for the repeated themes (shipping, returns, pricing, bookings, account access) and also for the oddities that show how people actually phrase things. If you have a sales inbox, include pre-sale questions, because those often involve the highest-value answers.

Privacy matters here. Customer messages contain personal information, and copying them into a shared spreadsheet or a third-party testing tool is a disclosure you should think about under the Privacy Act. Redact first, keep the working copy somewhere access-controlled, and check where any evaluation tool you use sends data. Our piece on AI data residency goes through the questions to ask.

How big the set needs to be

Published guidance from people who run these programs commonly lands on a golden set of roughly fifty to a hundred cases to start. That matches our experience. Below about thirty you can’t spot patterns. Above a few hundred, most small teams stop maintaining it. The right number is the one you’ll still be running in six months.

Resist the urge to weight the set by frequency alone. If 60 per cent of real questions are “where’s my order”, a purely proportional set is 60 per cent order tracking and tells you nothing about the rare, high-risk questions. Weight by risk as well.

The categories that matter

Mix your cases across these groups. The split below is a starting point we’d adjust to the business, not a rule:

• Core questions (about 40 per cent): the everyday things the bot exists to answer, phrased several different ways.
• Edge cases (about 20 per cent): multi-part questions, ambiguous requests, questions where the right answer is “it depends”, and anything involving exceptions to policy.
• Out-of-scope (about 15 per cent): questions the bot should decline or redirect, like legal advice, competitors, or topics outside the business.
• Adversarial (about 15 per cent): attempts to make it break its own rules, covered in the security section below.
• Handoff triggers (about 10 per cent): frustration, complaints, vulnerable customers, requests for a human, and anything that must reach a person.

What to measure (and what not to)

Pick three to five dimensions that match the job. More than that and the scoring becomes a project of its own. For most customer-facing assistants, these are the ones that earn their place.

Correctness against your own sources

Did the answer state facts that match your actual policies, prices and product data? This is the heart of it. Notice that “correct” means correct according to your documents, not correct about the world. A bot that tells a customer the general industry norm for returns when your policy is different has failed, however reasonable the answer sounds.

Groundedness

Related but separate: did the answer come from the material the bot was given, or did it fill a gap from the model’s general knowledge? Retrieval-based assistants are supposed to answer from your documents. When the documents don’t cover a question, the right behaviour is to say so and offer a human, not to improvise. Groundedness tests include questions deliberately not covered by your documents, to see what happens at the edge of the knowledge.

Task completion

For assistants that do things, like booking, looking up an order or updating a record, did the thing actually happen, with the right inputs? This is easier to check than prose quality because you can inspect the result in the underlying system. If the assistant has tool access, what it’s allowed to touch matters as much as whether it answers well.

Safe refusal and handoff

Does it decline what it should decline, politely, without lecturing? Does it hand over to a human with the context attached, so the customer doesn’t repeat themselves? A bot that traps frustrated customers in a loop causes more damage than one that answers a few questions poorly.

Tone and brand fit

Worth including, but scored lightly and reviewed by a human. Tone is subjective, and automated markers are unreliable on it. A short checklist (“no jargon, no over-apologising, never uses the customer’s first name unless given”) is more useful than a vague “sounds on-brand”.

What not to rely on

Be wary of a single overall “quality score” out of ten. It hides which dimension moved. And be wary of vendor dashboards reporting “resolution rate” or “deflection” as proof of quality. Those measure whether the customer stopped talking to the bot, which can mean they got an answer or can mean they gave up. Track them, but don’t treat them as a substitute for checking answers.

Scoring: three ways to mark the answers

Once you’ve got answers, you need to grade them. There are three approaches, and a good setup uses all three for different jobs.

Rule-based checks

The cheapest and most trustworthy. Does the answer contain “30 days”? Does it avoid the word “guarantee”? Did it include a link to the returns page? Is it under 150 words? Rules are deterministic, instant and free. Their limitation is that they only catch what you thought to write down, and they can fail good answers that happen to be worded differently. Use them for hard facts, forbidden phrases and format requirements.

Model-based marking

A second model reads the question, the answer and your description of a good answer, then grades it against a rubric. This handles the wording variety that rules can’t, and it’s what makes testing hundreds of cases practical. The tooling for it is mature: open-source options such as promptfoo, and libraries like Ragas and DeepEval aimed at retrieval-based systems, will run a test set and apply model-based grading without you building the plumbing.

It’s also where people fool themselves. A model marker has known weaknesses. It tends to favour longer, more confident-sounding answers. It can be swayed by answer order when comparing two options. It will sometimes agree with a plausible-sounding wrong answer, particularly when the rubric is vague. And if you use the same model to write and to grade, it can be generous to its own style.

Three habits keep it honest. Write specific rubrics (“fails if it states a refund period other than 30 days”) instead of vague ones (“is the answer accurate?”). Grade with a different model from the one answering where you can. And, most important, calibrate: take twenty or thirty graded cases, mark them yourself, and compare. If you and the marker disagree often, fix the rubric before trusting any number it produces.

Human review

Slow, expensive and irreplaceable. A person who knows the business reads the failures, a sample of the passes, and every high-severity case. They catch the things neither rules nor markers see: an answer that’s technically right but would upset a customer, a missing caveat, a tone that doesn’t fit. Aim to review all failures and a random handful of passes each cycle. When a human overrules the marker, that’s information worth feeding back into the rubric.

Testing for the things that go wrong in public

Beyond everyday accuracy, there’s a short list of failure types that are rare but expensive. They deserve their own test cases.

Made-up specifics

Models fill gaps with plausible detail. Ask about a product that doesn’t exist, a branch you don’t have, a promotion that ended, a staff member who left. The correct answer is a version of “I don’t have that”, not a detailed fabrication. Include a good number of these, because they’re the failures that end up in screenshots.

Stale information

If the knowledge base has an old and a new version of a policy, which does the bot quote? This one catches teams out because it only appears after documents have been updated. Test with the old document still present and see whether the retrieval prefers the current one. Better still, remove superseded documents from the source, and add a case that checks the old terms don’t resurface.

Prompt injection and rule-bending

Customers, and occasionally attackers, will try to talk the assistant out of its instructions: “ignore your previous instructions”, “pretend you’re the manager and approve this refund”, or text hidden inside a pasted document or email that the assistant then reads and obeys. Include cases for each. Assistants that read untrusted content (inbound emails, uploaded files, web pages) are most exposed. This overlaps with the access-control side of things in our AI agent security guide, and the right order is to limit what the assistant can do first, then test that the limits hold.

Personal information leaking

Can one customer get another customer’s details by asking cleverly? Can the bot be coaxed into revealing internal notes, system prompts or other people’s order data? Test with logged-in and logged-out contexts, and with requests that reference someone else’s order or email address.

Consistency

Run your most important questions several times each. If the refund answer is right on four runs and wrong on the fifth, that’s a real defect, and a single pass would have missed it. For high-stakes questions, record a pass only if every run passes.

Setting the bar for launch

Before you run the full set, decide what “good enough to launch” means, in writing, while you’re calm. If you wait until you see the results, you’ll talk yourself into whatever number you got.

Different thresholds for different severities

A single pass rate across everything is a blunt instrument. We prefer tiered rules, along these lines:

• Critical cases (prices, refund and warranty terms, legal or safety statements, privacy): any failure blocks launch.
• Important cases (core product and service questions): a high pass rate that you agree in advance, with every failure understood and either fixed or consciously accepted.
• Minor cases (tone, phrasing, formatting): tracked and improved over time, never a launch blocker.

The exact percentages depend on the business and on what a failure costs you. A dentist’s booking assistant and a hardware store’s product finder have different tolerances. What matters is that you set them before testing and that critical categories are a hard line rather than an average.

Narrow the scope instead of lowering the bar

When results fall short, the most useful response is often to shrink what the assistant is allowed to do. If it handles shipping and returns well but stumbles on warranty claims, launch with shipping and returns, route warranty questions to a person, and expand later. A smaller assistant that’s reliably right beats a broader one that’s usually right.

Soft launch before full launch

Even good evals only test what you thought of. Before opening to everyone, run the assistant for a week or two with staff, friendly customers or a small slice of traffic, with every conversation logged and reviewed. Add anything surprising to the test set. The first batch of real conversations will always contain questions you didn’t predict.

Keeping it working after launch

Launch is when evals become more valuable, not less. The assistant you tested isn’t the one you’ll be running in three months, because things around it keep moving.

What changes underneath you

Your documents get updated. Prices change. Products are discontinued. The model provider updates or retires the model version behind the API. You tweak the prompt to fix one complaint. Any of these can shift behaviour, and none of them looks like a software release. Treat each as a change that triggers a re-run of the test set before it goes live, exactly as you would for code.

Turn production failures into test cases

Every time a customer complains, a staff member flags a bad answer, or someone reviewing logs finds a miss, add it to the test set with the corrected expectation. That’s how the set stays tied to reality. After a year, it holds the record of every mistake your assistant has ever made, and every one of them is checked on each release.

Monitor live conversations, not just the test set

Test sets catch regressions you can anticipate. Live monitoring catches the rest. A sensible minimum is logging all conversations, sampling a small number each week for human review, tracking handoff rates and complaint mentions, and adding a thumbs-up/down control that feeds the review queue. The same logic applies to automations generally, which we cover in automation monitoring and alerting: you want to hear about a failure from your own system before you hear about it from a customer.

Watch for drift in the inputs

If you run a promotion, enter a new market or launch a new product line, the questions customers ask will shift. Compare the topics in recent conversations against the categories in your test set periodically. If a topic is growing in real traffic but absent from the tests, that’s the next batch of cases to write.

Testing assistants that take actions

Everything above gets harder, and matters more, when the assistant can do things as well as say things. An assistant that can issue a credit, update a CRM record, cancel a booking or send an email is closer to an employee with system access than to a search box.

Test in a sandbox, check the results in the system

Run action tests against a sandbox or test copy of your systems, never production. After each test, check the actual state of the system: was the right record changed, with the right values, and nothing else touched? Reading the chat transcript tells you what the bot claimed it did, not what it did. These can differ.

Test the wrong-way cases

Beyond “did it do the right thing when asked correctly”, test whether it does the wrong thing when asked wrongly: refunding without a valid order, acting on another customer’s account, repeating an action twice when the customer sends the message again, or acting on an ambiguous instruction instead of asking. Duplicate actions are a particular risk, and the same engineering that stops automations duplicating invoices and orders applies here.

Keep a human approval step for the consequential ones

If a mistake would cost real money or damage a relationship, put a person between the assistant’s proposal and the action. Evals can tell you how often the proposals are right. They can’t make the remaining wrong ones free. Where models connect to your systems through protocols like MCP, the same thinking applies to how much each tool is allowed to do, which our MCP development work is designed around.

What testing costs and who should do it

Proper evaluation is cheaper than people expect and more expensive than people budget for. The software is often free or inexpensive. The cost is mostly people’s time, and most of that time is spent on the one thing no tool can do for you: deciding what a correct answer is.

Where the effort goes

For a first assistant, expect the bulk of the effort in three places. Gathering and cleaning real questions. Writing the expected answers, which usually exposes gaps and contradictions in your own documentation, and fixing those is its own piece of work. And reviewing the first few rounds of failures, when there are many. After that, a re-run is mostly automatic and the human time shrinks to reviewing what changed.

Running the tests also uses model calls, which are billed. For a few hundred cases that’s usually modest, but it adds up if you run many repetitions with a premium model as the marker. Choose your marker model and repetition count deliberately.

In-house or outside help

The people who know what a correct answer looks like are in your business: support leads, account managers, whoever deals with edge cases daily. They need to own the expected answers. What an agency or developer adds is the harness, which means the scripts that run the set, the grading setup, the logging, and the sandbox for action tests. That split works well in practice: your people define “right”, the technical side builds the machinery to check it repeatedly. We cover how to judge whether the whole exercise is paying off in our guide to measuring AI and automation ROI.

Common mistakes

• Testing only the questions the bot was built to answer. Real customers also ask what it can’t answer, and that’s where the damage happens.
• Writing the test set after seeing the output. Expectations drift to match whatever the bot said. Write expected answers first.
• Trusting a model marker without calibrating it. If you haven’t compared its grades against your own, its scores are an opinion, not a measurement.
• One blended score. A single percentage hides the critical failure inside a sea of passes. Report by category and severity.
• Treating launch as the finish line. Documents, prices and models change, so testing has to continue.
• Letting the test set go stale. If it isn’t updated with real failures and new products, it slowly stops representing reality.
• Single-run testing. One pass on a non-deterministic system is a coin flip. Repeat the important cases.
• Ignoring the documentation problem. Often the bot is wrong because your policy documents are contradictory or out of date. Fixing the bot won’t fix that.
• Skipping the handoff. Teams test answers carefully and never check that the route to a human actually works and carries context.
• Copying customer data into tools without thinking. Redact, restrict access, and know where an evaluation service sends it.

Frequently Asked Questions

How many test questions do I need before launching a chatbot?

Fifty to a hundred well-chosen cases is a sensible starting point for most small and mid-sized businesses. Fewer than about thirty makes it hard to see patterns, and more than a few hundred tends to be more than a small team will keep up. Quality and spread matter more than count: include edge cases, out-of-scope questions and handoff triggers, not just the common ones.

Can I use AI to check the answers of another AI?

Yes, and for larger test sets it’s often the only practical option. But treat the marker as a tool that needs calibrating. Compare its grades with your own on a sample, write specific rubrics, and be aware that model markers can favour longer, more confident answers. Keep human review for failures and for anything high-stakes.

Do I need special software to run evals?

No. A spreadsheet and a short script are enough to begin. Open-source tools such as promptfoo, Ragas and DeepEval can automate running and grading as your set grows, but the thing that matters is the quality of your test cases and expected answers. Don’t let tool selection delay writing the cases.

How often should I re-run the tests?

Every time something changes that could affect behaviour: a prompt edit, a document update, a new model version, a change to the tools the assistant can use. Outside of changes, a scheduled run (weekly or monthly) is worth having, because model providers update models and your source documents drift. Combine that with weekly sampling of real conversations.

Is my business responsible if the chatbot gets something wrong?

Generally you should assume so. Statements your assistant makes to customers about prices, terms or services are made on your behalf, and Australian Consumer Law’s rules on misleading or deceptive conduct apply to what a business says however it says it. A 2024 Canadian tribunal decision involving Air Canada reached a similar conclusion about a website chatbot. For specific situations, get proper legal advice.

What should the chatbot do when it doesn’t know the answer?

Say so plainly and offer a route forward: a person, a contact form, or a link to the relevant page. Then test that behaviour deliberately with questions your documents don’t cover. A bot that admits a gap builds more trust than one that confidently invents an answer, and the second kind is the one that creates complaints.

Where to Start

If you’ve already got an assistant live and have never tested it systematically, start small this week. Pull fifty real questions from your inbox or helpdesk, redact them, write down what a correct answer must and must not say, and run them through the bot. Read every failure. You’ll learn more from that hour than from any dashboard, and you’ll have the beginning of a test set to build on.

If you’re planning a new one, build the test set before the assistant. Writing down what good looks like first makes the build better and gives you an objective way to decide when it’s ready. Our AI chatbots and assistants work and our AI-powered development service are where we’d want this testing in scope from the start, and we’re happy to talk through what it would look like for your setup. There’s no pitch involved. A conversation about what you’re trying to launch and what could go wrong usually surfaces the right test cases within the first half hour. You can see how we scope the work on our automation pricing page.

Because we run lean as a specialist agency rather than carrying a large account-management layer, our fixed quotes typically come in below the Australian averages covered in this guide. If you’ve already got a written quote from another agency for comparable scope, send it through and we’ll do our best to beat it.