AI Engineering · 7 MIN
How to choose an AI chatbot development company
How to choose an AI chatbot development company: test grounding in your content, scoring on real conversations, human handoff, channels and data handling.
Customers treat whatever your chatbot says as your company's word, so the builder you hire is really building a spokesperson. Choose an AI chatbot development company that answers from your approved content instead of the model's general knowledge, tests on transcripts from your real customers, hands off to a person cleanly when it should and can explain where your conversation data goes. A slick demo on a sample website proves almost none of that. This guide covers how to judge those four areas, when an off-the-shelf platform beats a custom build, and what to put in the contract.
- A useful business chatbot answers from your own documents and data, and says so when it does not know.
- Test vendors on a set of real past conversations with answers your team has approved, not on demo questions.
- A clear handoff to a human, with the conversation history attached, matters as much as the bot's answers.
- Ask where conversation data is stored, which model providers process it and whether it is used for training.
- Off-the-shelf chatbot platforms suit simple FAQ coverage. Custom builds suit bots that need your systems, your rules or your languages.
- Nactore is a technology and growth company that builds and ships AI chatbots and the support, product and website systems they plug into.
What should a business chatbot actually do?
Start from the job. Most business chatbots do one of four things: answer product and policy questions, help customers with their own account (orders, bookings, tickets), qualify and route sales leads, or help staff search internal knowledge. Each needs different access and different safeguards.
Write down which one yours is, who uses it, which systems it needs to read and what it must never say or do. A bot that only answers public questions is a much smaller project than one that looks up order status or changes a booking. The second kind starts to behave like an agent, and our guide to choosing an AI agent development company applies too.
How do you check it answers from your content?
The core technique is retrieval: the system finds the relevant passages in your help center, policies or product data and asks the model to answer from them. Ask each vendor how they do this and how they handle the hard parts.
- Stale content. How are documents updated, and how fast does the bot reflect a policy change?
- Conflicting sources. What happens when two documents disagree?
- No answer found. Does the bot say it does not know and offer a person, or does it guess?
- Citations. Can it show the customer, or your team, which source an answer came from?
If a vendor proposes training a custom model instead, ask why. For most chatbots, retrieval over your content is simpler to update and easier to audit. Our comparison of RAG and fine-tuning explains the choice.
How should you test chatbot vendors?
Give every vendor the same test. Pull a sample of real past conversations, remove personal data, and have your team write the correct answer for each. Include the awkward ones: angry customers, questions outside your scope, two questions in one message and attempts to make the bot say something it should not.
Ask each vendor to run their approach against that set and report how many answers your team would approve, how many were handed off and how many were wrong. Our guide on how to evaluate LLM output quality explains how to score this without a data science team.
Include a few questions where the right answer is "I can't help with that, here is a person who can." A chatbot that answers everything confidently is more dangerous than one that knows when to stop.
Ask about manipulation as well. A public chatbot will meet users who try to override its instructions, which OWASP lists as prompt injection, the first risk in its list for LLM applications. Good vendors can show the guardrails they add and how they test them. See what guardrails an LLM feature needs.
Platform or custom build?
Chatbot platforms have improved and are often the right first step. A custom build earns its cost when the bot needs deeper access or stricter control.
| Need | Off-the-shelf platform | Custom build |
|---|---|---|
| Answer public FAQs from a help center | Usually enough | More than you need |
| Look up orders, bookings or accounts | Limited to the platform's connectors | Built against your own systems and permissions |
| Strict rules on tone, claims and topics | Basic settings | Your own guardrails, tested on your cases |
| Several languages or dialects | Varies by platform | Model and tests chosen per language |
| Control over data location and retention | Set by the platform's terms | Set by you |
| Measurement against your own test set | Rarely built in | Part of the build |
A good chatbot development company will tell you honestly when a platform fits. If every vendor you speak to recommends a custom build for a simple FAQ bot, keep asking. For non-English bots, the details in what to know before building an Arabic AI chatbot show how much language choice changes the work.
What about handoff, channels and reporting?
The handoff to a human is where many chatbots lose customers. Ask how the bot decides to hand off, whether the agent receives the full conversation and whether the customer has to repeat themselves. If you already run a support desk, the bot should create or update tickets there, as described in AI support ticket triage.
Then confirm channels (website, mobile app, WhatsApp, email, internal tools) and reporting. You want to see which questions come up most, which ones the bot failed and what gets handed off, so your team can fix the content behind the failures.
What belongs in a chatbot contract?
- Data handling. Where conversations are stored, for how long, which model providers process them and whether any of it is used for training.
- Ownership. You own the code, prompts, content index, test set and conversation logs.
- Acceptance. The score on your test set that the bot must reach before it goes live.
- Run costs. Expected cost per conversation, with a cap.
- Maintenance. Who updates content, reviews failures and retests after a model change.
Frequently asked questions
How do we stop a chatbot from making things up?
Have it answer only from retrieved passages of your approved content, tell it to say when it does not know, and test that behavior on questions with no answer in your documents. No method removes errors entirely, which is why handoff and review of failed conversations matter.
Who is responsible if the chatbot gives a customer wrong information?
Assume your company is, since customers see the bot as speaking for you. Ask your counsel about your own jurisdiction, and keep the bot's scope narrow on topics like refunds, pricing and legal terms.
Do we need a chatbot on WhatsApp or only on our website?
Put it where your customers already ask questions. Each channel adds integration and testing work, so start with the busiest one and add others once the answers are reliable.
How do we know if the chatbot is working after launch?
Track how often conversations are resolved without a person, how often they are handed off and what customers rate them. Review a sample of transcripts every week and turn failures into new test cases.
A chatbot is only as good as its last bad answer
The right chatbot partner spends more time on your content, your test conversations and your handoff than on the chat window. Judge vendors on how they handle the questions the bot should not answer, and the rest of the evaluation gets easier.
Thinking about a chatbot for support, sales or your own team? Book a free 30-minute call and bring a few real customer questions.
Want to apply this to your business?
Book a free 30-minute call. We will tell you what we would do first.