AI Engineering · 8 MIN
How to catch your AI chatbot giving customers wrong answers
Hand review cannot keep up with a customer-facing AI bot. Nightly automated checks, called evals, score every chat and flag problems before customers complain.
The way to catch an AI chatbot saying something wrong before a customer complains is to check every conversation automatically, every night, against rules written for your business. These automated checks are often called evals, short for evaluations. An eval is a repeatable test that scores what the AI said against a clear definition of a good answer. This guide covers what counts as a failure, how the checks work, the morning report, cost, and a two-week setup plan.
- Reading a sample of chats by hand misses rare, costly mistakes.
- Define failure first: unverified promises, wrong facts, wrong tone, data leaks, and policy breaks.
- Score every conversation nightly with rules plus a second model, and run tricky test cases on every change.
- A one-screen morning report with one named owner turns flags into fixes.
- Nactore builds, ships, and measures customer-facing AI as one partner, with answer checks from the first release.
Why does checking chats by hand stop working?
Checking AI chats by hand stops working because a person can only read a small sample, and the answers that hurt you are rare, so a sample usually misses them. Hand review is also slow and costly per conversation, and it gets skipped on busy days, when the bot carries the most load.
A reviewer also cannot tell whether a refund the bot confirmed actually went through without opening another system. So hand review catches odd tone and misses confident wrong facts, the ones customers act on.
A sample tells you how the bot usually behaves. Customers complain about how it behaved once.
What counts as a failure for your business?
A failure is any answer your business would not want a customer to act on. Write these down as named categories before building any check, because a check can only catch what someone defined. For a customer-facing bot in finance, health, or transport, five categories cover the ground that matters first.
| Failure type | Example | How the check catches it |
|---|---|---|
| Unverified promise | "Your refund will reach you by Friday" with no refund record | A rule flags promise phrases, then looks for a matching tool call |
| Wrong fact | Says a bus is "arriving now" when live data shows it late | Compares the claim with the data the bot fetched |
| Wrong tone | Makes light of a failed payment or missed appointment | A second model grades tone against a short rubric |
| Data leak | Reads out another customer's phone or card number | Pattern rules, plus a check that every record shown belongs to the logged-in user |
| Policy break | Gives dosage or investment advice, or waives a fee | A second model checks the reply against a list of topics the bot must refuse or hand to a human |
Give each category an owner and a severity. A data leak might alert someone at once. A tone slip can wait for the morning report.
Make the bot look up real data before it answers
The cheapest failure to fix is the one that never happens. Wrong facts often come from a bot answering from memory instead of asking your systems. Give it tools that fetch the live booking, balance, appointment, or vehicle position, and require that any time-sensitive or account-specific claim comes from a tool result in the same turn.
This also makes checking easier. When every claim sits next to a tool result in the log, the nightly check can compare what the bot said with what the data said. A factual claim with no tool call behind it is worth flagging on its own. See our guide to guardrails in production.
How do the automated checks work?
Automated answer checks run in two loops. A nightly job scores every conversation from the previous day using simple rules plus a second AI model acting as a grader. A small set of known tricky test cases runs on every change, so a fix in one place does not quietly break another.
Score every conversation nightly
Rules go first because they are cheap, fast, and consistent. Think pattern matches for phone and card numbers, a list of promise phrases, and a flag for any time or amount stated without a tool call behind it.
Then a second model reads each conversation with a short rubric, one question per failure category, and returns pass or fail per category, the sentence at fault, and a one-line reason. Use a different model from the one running the bot, so the grader is less likely to share its blind spots. Grade a sample by hand at the start and adjust the rubric until you agree.
On a production support agent we know closely, at a transport company, the nightly check flagged the agent telling riders their bus was "arriving now" when it was 29 minutes late. The vehicle data sat in the same log as the reply, so the mismatch was plain.
Run tricky test cases on every change
Keep a small set of known hard conversations, often called a golden set, such as the late bus, the failed refund, or the customer asking for someone else's booking. Every prompt, model, or tool change runs this set before it ships, and if a case that used to pass now fails, the change waits. Each real failure the nightly check finds becomes a new case. More in why AI features need evals before production.
Write each test case as a real customer message, typos and all, with a one-line note on what a good answer must and must not contain. Tidy, invented messages hide the problems real ones expose.
What should the morning report contain?
A useful morning report fits on one screen and tells one named person what to do today. It shows yesterday's conversation count, failures by category against the day before, and the worst few conversations with the exact sentence quoted and a link to the chat.
- One owner. Name the person who reads it. A report posted to a shared channel belongs to nobody.
- Severity first. Data leaks and policy breaks at the top, tone at the bottom.
- Quote, do not summarize. Show what the bot said next to the data it should have used.
- An action per item. Each flag ends in one of four outcomes: fix the prompt, fix a tool, add a test case, or mark it a false alarm.
False alarms show where the rubric needs tightening. A report that cries wolf soon stops being read.
How do you keep the cost low?
Automated checks stay cheap when rules do most of the filtering and the grading model is modest, batched, and run once a night instead of live. In one production system, a manual review costing about $40 a day was replaced by a nightly check that scores every AI conversation for under half that cost.
- Rules before models. Code checks cost almost nothing to run.
- A smaller grader. A narrow rubric rarely needs your largest model. Test a smaller one against your hand-graded sample.
- Batch overnight. Several model providers offer discounted rates for jobs that can wait a few hours.
- Cache the rubric. The grading instructions repeat for every conversation, so prompt caching trims the repeated part.
Your first two weeks
- Days 1 to 2, write the failure list. Name your categories and give each a real example, a severity, and an owner.
- Days 3 to 4, fix the logs. Store every conversation with the tool calls and results the bot used, or the checks have nothing to compare against. See what to log.
- Days 5 to 6, add the rules. Promise phrases, personal data patterns, account ownership, and claims with no tool call behind them.
- Days 7 to 8, add the grader. Write the rubric, run it on a few days of past conversations, and grade the same sample by hand. Adjust until you agree.
- Days 9 to 10, build the golden set. Start with the worst real conversations from steps 1 and 4, plus a few attempts to trick the bot, and run it on every change.
- Days 11 to 12, ship the morning report. One screen, one owner, delivered before the working day starts.
- Days 13 to 14, review and trim. Go through the false alarms, tighten the rubric, and compare the running cost with hand review.
Frequently asked questions
Can the grader model be wrong?
Yes. It can miss failures and flag good answers. That is why rules handle anything code can check, why you calibrate the grader against hand grading, and why the report lets someone mark a false alarm.
Should the checks run live instead of nightly?
Run the cheap rules live where a failure would cause real harm, such as personal data in a reply. Keep model grading nightly, where it costs less and does not slow replies.
Do we still need people reviewing chats?
Yes, on a better job. People read the flagged conversations and decide what to fix, instead of hunting for problems in a random sample.
The takeaway. A customer-facing bot will say something wrong at some point. What you control is whether you hear about it from a morning report or from a customer. Nactore is your embedded tech and growth partner, and we own the outcome, not just the build. We set up automated answer checks for customer-facing AI and hand over a morning report your team can act on. Write to us at hello@nactore.com.
Want to apply this to your business?
Book a free 30-minute call. We will tell you what we would do first.