Buyer Guides · 7 MIN
Red flags when hiring an AI engineer: a 10-point checklist and what to ask instead
Red flags when hiring an AI engineer or AI company: a 10-point checklist, the question to ask instead for each, and what a good answer sounds like.
The biggest red flags when hiring an AI engineer are a portfolio of demos with nothing in production, no way of measuring output quality, vague answers about failure, and a pitch built around a model name instead of your problem. The same flags apply to an AI development company. One flag deserves a follow-up question, and several should end the conversation. Below is a 10-point checklist with the question to ask instead and what a good answer sounds like.
- Demos are cheap. Ask for systems running in production and how they are monitored.
- Anyone serious about AI can explain how they measure output quality.
- Honest engineers can name how a system fails and what it should not be used for.
- For a company, add bench swaps, early guarantees, and unclear ownership of code, prompts, and eval sets.
- Nactore, a technology and growth company, expects buyers to ask these questions and answers them with evals and production evidence.
What are red flags when hiring an AI engineer?
The red flags when hiring an AI engineer are a portfolio of demos with nothing in production, no method for measuring output quality, no honest story about a system that failed, a model chosen before your problem was understood, and no sense of what each request costs. Each one can be tested with a single question, listed in the checklist below. Strong engineers answer with specifics and numbers. Weak ones answer with adjectives.
What are red flags when hiring an AI development company?
The red flags when hiring an AI development company include everything above, plus risks from the contract and team structure.
- Senior people sell, juniors deliver. Ask to meet the engineers who will write the code, and get their names in the contract.
- Guarantees before seeing your data. A fixed accuracy number before discovery hides risk. Ask what they need to see to estimate.
- No acceptance criteria. Ask what written pass bar, measured on what test set, decides that the work is done.
- Unclear ownership. Ask who owns the code, prompts, eval sets, and fine-tuned weights when the engagement ends.
- Lock-in by design. Ask how you would run and change the system without them.
Whether to use a partner at all is covered in in-house AI team vs AI engineering partner, and contract shape in fixed-scope AI pilot vs time and materials.
Red flag checklist: what to ask instead
Language models make an impressive demo an afternoon's work, so a prototype builder and an engineer who can run a system for years look alike on paper. Use this checklist in the call. Tick a flag when the answer has no specifics.
| Red flag | Ask instead | A good answer includes |
|---|---|---|
| 1. Portfolio is only demos | "Which of these run in production today, and who uses them?" | A named live system, its users and how it is monitored |
| 2. No measurement of quality | "How do you score output, and on what data?" | An eval set, a metric and scores over time |
| 3. Cannot describe failures | "Show me a case where it failed and what you changed." | A specific incident, how they found it and the fix |
| 4. Leads with a model name | "Why this approach for our workflow?" | Questions about your data before any tool choice |
| 5. Promises certainty | "What error rate should we plan for?" | A range, and what happens when it is wrong |
| 6. Guarantees results before seeing data | "What would you need to see to estimate this?" | A sample of your data and a short test first |
| 7. Vague on data and IP | "Where does our data go, and who owns the outputs?" | Named providers, retention terms and your ownership |
| 8. No monitoring plan | "What do you log, and who gets alerted?" | Logs, alerts and an owner after launch |
| 9. No cost awareness | "What does each processed item cost?" | A per-request cost and latency figure |
| 10. Bench swap after signing | "Who exactly will work on this, by name?" | Named engineers you can interview |
For live walkthroughs, see how to evaluate an AI demo.
How do you test the three flags that matter most?
Have they shipped to production?
People who have run production systems mention retries, rate limits, alerting, cost caps, and rollbacks without prompting. Ask them to walk through their last incident after launch: how they found out and what they changed. Blaming the model, or having no story, is a flag. Our prototype to production checklist lists what a good answer covers.
Do they measure output quality?
This is the most useful filter. A team that ships AI without scoring output is guessing, and so are you. Ask to see an eval set, the metric, and how scores moved over time. Strong teams treat AI evals before production as core work and can explain which grader fits which output, as covered in how to evaluate LLM output quality.
A simple screen: ask "how will we know when it is good enough to ship?" If the answer is a feeling, a demo, or "when you are happy," keep looking.
Are they clear on data, security, and ownership?
The red flag here is an evasive answer. A capable team can explain which providers touch your data, what is retained, and who owns the code, prompts, and eval sets. Use IP and data terms for AI projects as a checklist, and have your counsel review the actual terms.
How should you run the vetting conversation?
- Start with your problem, not their pitch. Describe a real workflow and see if they ask about your data.
- Ask for a production example. Probe for failure stories and monitoring.
- Request the measurement plan. Ask how they would test on your data.
- Meet the people doing the work. Interview the engineers, not only the sales lead.
- Run a bounded first step. A fixed-scope pilot with a written pass bar reveals more than any interview. See how to scope an AI pilot.
Good signs are the mirror image: questions about your data first, success defined on a test set, and candor when a use case is poorly suited to AI. For country-specific checks, see our guides to hiring an AI company in the USA, evaluating one in India and choosing one in the UAE.
Frequently asked questions
What is the biggest red flag when hiring an AI engineer?
No way to measure output quality. An engineer who cannot explain how they would score results on your data cannot tell you whether a change helped or hurt, so every other claim rests on a guess.
Is a strong portfolio enough to trust an AI team?
No. A portfolio shows what was built, not how reliably it runs. Ask about production use, monitoring, and failures.
Do the same red flags apply to other AI roles?
Yes. For product managers, data scientists and AI leads, the shared red flag is someone who cannot say how success will be measured. Ask every AI candidate how you will know the work succeeds, and how you will know when it stops working.
How is vetting an AI developer different from vetting an AI development company?
The core questions are the same. For an individual, also ask about continuity and who covers when they are unavailable. For a company, add who does the work, who owns the outputs, and how you would run the system without them.
What is the fastest way to reduce hiring risk?
A short, fixed-scope pilot with a written pass bar. It shows how the person or team works with your data and gives you evidence before a larger commitment.
Hire for proof, not polish
Look for production systems, measured quality, candor about failure, and clear answers on data and ownership. Those separate strong engineers and partners from impressive demos. For the full hiring process, read how to hire AI developers. Nactore holds itself to the same bar as one partner that builds, ships, ranks, and grows, measured on your growth. See what we do.
Want this built for your team? Book a free 30-minute call.
Want to apply this to your business?
Book a free 30-minute call. We will tell you what we would do first.