Agents and MCP · 7 MIN
How to choose an AI agent development company
How to choose an AI agent development company: check tool permissions, task-level evals, trace logging, approval steps and cost per task before you sign.
An AI agent takes actions in your systems, so choosing who builds one is closer to hiring for production infrastructure than for a chatbot. Pick the company that can show you three things in detail: how it limits what the agent is allowed to do, how it measures whether tasks were completed correctly, and how it records every step so a failure can be traced. Polish, model names and demo speed matter far less. Below are the questions that separate agent builders from chatbot builders, a scoring table, and the contract terms to insist on.
- An agent reads, decides and acts across your tools. The risk sits in the actions, so permissions and approvals come first.
- Many "agent" projects are better built as fixed workflows. A good vendor will tell you when yours is one of them.
- Ask for evals scored on whole tasks, not single answers, using your real cases.
- Every run should leave a trace you can read: inputs, tool calls, results and cost.
- OWASP lists excessive agency as a named risk for LLM applications. Use its three causes as interview questions.
- Nactore is a technology and growth company that builds and ships agents, the software they act on and the product around them.
What makes agent development different from other AI work?
A chatbot produces text that a person reads. An agent produces actions: it updates a CRM record, issues a refund, books a meeting or files a ticket. A wrong answer in a chatbot is embarrassing. A wrong action in an agent can cost money or leak data before anyone notices.
That changes what you are buying. You need a vendor that designs tools the model can call correctly, scopes permissions tightly, logs every step and puts a person in the loop where the stakes justify it. Our guide on how to design tools for AI agents shows what good tool design looks like in practice.
Does your use case need an agent at all?
Ask this before you compare vendors. If the steps are known in advance and rarely change, a deterministic workflow with a model at one or two decision points is cheaper, faster and easier to test. Agents earn their place when the path depends on what the system finds along the way.
A strong vendor will raise this question themselves. If every proposal you receive calls for an agent, ask each one what a workflow version would look like and why they rejected it. Our comparison of AI agents and workflows gives you the criteria. The same skepticism applies to proposals with several cooperating agents, covered in when multi-agent systems are worth it.
What should you ask about permissions and safety?
The OWASP Top 10 for LLM applications names "excessive agency" as a risk and traces it to three causes: too much functionality, too many permissions and too much autonomy. Turn each into a question.
- Functionality. Which tools will the agent have, and why does it need each one? Unused tools should not exist.
- Permissions. Does each tool run with the narrowest access possible, scoped to the user it acts for?
- Autonomy. Which actions require a human to approve before they happen, such as payments, deletions or messages to customers?
Ask about indirect prompt injection too. OWASP describes prompt injection that arrives through external content such as websites or files. An agent that reads emails or web pages needs a plan for instructions hidden in that content. If your agent connects through MCP, the vendor should also know the issues in MCP security risks.
How do you judge their evals and observability?
Agent quality is measured per task, not per reply. Ask how they will score whether the agent finished the job correctly, used the right tools in a sensible order and stopped when it should have.
| What to check | Weak answer | Strong answer |
|---|---|---|
| Task evals | "We tested it and it works" | A set of your real cases with expected outcomes, scored on completion and correctness |
| Failure cases | Only happy paths shown | Planted edge cases, refusals and partial data in the test set |
| Tracing | Logs of final answers only | A readable trace of every model call, tool call, input, output and cost |
| Approvals | "The model is very accurate" | Named actions that pause for a human, with an audit record |
| Cost | A monthly estimate with no basis | Cost per task measured on the eval set, plus a cap |
| Rollback | Not discussed | Kill switch, read-only mode and a way to reverse actions |
For what a trace should contain, see what to log for LLM observability. For approval design, see human-in-the-loop automation.
Ask the vendor to show you a real trace from an agent run that went wrong, with the cause they found. A team that has run agents in production has one ready. A team that has only built demos will change the subject, which is the first item on our AI engineer red flag checklist.
How should you run the selection?
- Write the task as a job description. What the agent does, which systems it touches, what it must never do and who approves what.
- Collect real cases. A few dozen historical examples with the correct outcome, including messy ones.
- Send all vendors the same brief. Ask for their approach, tool list, approval points and eval plan in writing.
- Score the written answers. Use the table above. Discard any proposal that skips permissions or tracing.
- Fund a small paid build. One narrow task, run against your cases in a test environment, with a pass bar agreed before work starts.
- Decide on evidence. Compare completion scores, traces and cost per task, not the polish of the demo.
What should the contract say?
- Ownership. You own the code, prompts, tool definitions, eval sets and traces.
- Access. Credentials live in your accounts, scoped per tool, and are revoked at the end.
- Acceptance. The agreed task score is the definition of done.
- Run costs. Model and infrastructure costs are estimated per task and capped.
- Incidents. How fast they respond when the agent takes a wrong action, and who pays for cleanup.
Our general checklist for choosing an AI agency covers the broader vendor terms.
Frequently asked questions
What is the difference between an AI agent company and an AI chatbot company?
A chatbot company builds systems that answer questions. An agent company builds systems that take actions in your software, which demands tighter permissions, approvals, task-level testing and full traces. Some vendors do both well, but check the agent-specific skills separately.
Should an AI agent run fully autonomously?
Rarely at first. Start with human approval on any action that is costly, irreversible or customer-facing, and remove approvals only where the traces show the agent is reliably correct.
Which framework or model should the vendor use?
The vendor should choose based on tests on your cases, and explain the choice. Be wary of a vendor that picks a framework before seeing your task, or that cannot swap models without a rebuild.
How do we keep agent costs under control?
Ask for cost per task measured on the eval set, a hard cap per run, and caching where the same context repeats. Our note on prompt caching explains one of the main levers.
Choose the builder who limits the agent
The best agent builders talk more about what the agent cannot do than about what it can. Choose the one whose permissions are tight, whose evals use your real tasks and whose traces you can read line by line.
Planning an agent that touches real systems? Talk it through with us in a free 30-minute call.
Want to apply this to your business?
Book a free 30-minute call. We will tell you what we would do first.