What is the checklist for taking an AI prototype to production?

A prototype becomes production when it has measured quality, visible failures, a known cost, safe infrastructure, a review gate, and a habit of checking facts before reporting them. This checklist comes from failures we actually hit, not from a template. Use it before launch, and treat every unchecked line as a decision you are making on purpose.

What separates a prototype from production?

A prototype answers "can this work?" Production answers "what happens when it does not?" The gap is rarely the model. It is everything that makes failure visible, bounded, and recoverable. The checklist below is ordered by what hurt us most in practice.

1. How do you prove quality before launch?

Build an evaluation set before launch and report three numbers separately. Right, wrong, and no answer. A wrong answer that sounds confident is a different failure from an abstention, and they cost the user differently. We describe the full approach in an attribution engine with evals and the general method in AI evals before production.

• Say what the set covers. A small set gives a number about that set. Never quote it as a general rate.
• Allow abstention. If the system cannot say "I do not know," it will guess.
• Re-run on every change. Diff per-case results, not only totals.

2. Where do failures go?

They go nowhere unless you build the path. In one production build, Django logged no tracebacks with DEBUG=False because, per the Django logging documentation, the default configuration only displays log records when DEBUG=True. Every other bug became harder to find.

1. Configure logging first. Send application and framework logs to stdout at an error-inclusive level before the first deploy.
2. Log third-party failure bodies. A bare status code often says nothing. A mail API returned a bare 403 until we logged the response body.
3. Set timeouts and retries on model calls. Decide what the user sees when a call fails or is slow.
4. Record model inputs and outputs in a way you can search. See LLM observability: what to log.

3. What does the model call actually cost?

Measure it as a share of what one successful transaction earns, before you optimize. In one paid product the model cost was a tiny fraction of each sale, and the real constraint was acquiring customers. If your ratio looks like that, spend effort elsewhere. If your ratio is large, start with LLM cost control and prompt caching.

Also check the settings that quietly change cost. Reasoning modes can be on by default and bill as output tokens, as covered in LLM settings for temperature and cost. Decide per call whether the task needs them.

4. What infrastructure traps should you check?

These are the ones that took things down for us.

• Quota-capped free tiers. A serverless database with an automatic usage quota refused to start after the quota ran out, and the whole stack went down with it. Production should not rest on a dependency that can be switched off by usage.
• Idle pauses and cold starts. Free tiers can pause or sleep when idle. Test the first request after a quiet period.
• Address families. A database host reachable from a laptop can be unreachable from the app host if it is IPv6-only. Use the pooler or an IPv4 endpoint.
• Vendor SDKs. One payment SDK failed to import on a modern runtime. When the protocol is small, plain HTTP and the standard library can be less code. See what breaks when you ship an AI product with payments.
• Env parsing. Our env reader treated the whole line as the value, so an inline comment broke a flag silently.
• Secrets. Live keys only in the production host's environment, never in the repository, and public values only in client bundles.
• Firewall and crawlers. A firewall rule returned errors to AI crawlers while serving Googlebot normally. Test with each crawler's user agent, as covered in Cloudflare for AI products.

5. How do you gate interface changes?

Do not judge an interface from its code. We had a first version of an owner console rejected by the founder, and a review of real screenshots at desktop and phone widths found problems that code review had missed, including a stray typeface from marketing styles and global CSS collisions that only appear on screen.

Since then, no interface ships without a PASS from a design reviewer who screenshots the real build at 1440 and 390 pixel widths and scores it against a rubric. The loop is fix, re-review, repeat until pass. In our case, one console went through several rounds before it passed. Scoping the new CSS under its own container, with explicit resets, stopped styles from leaking in from other parts of the site.

6. How do you stop reporting wrong facts?

This one is about people, and it cost us credibility twice. Both times, one command and one glance became a claim.

• A shell pipeline bound tighter than intended and reported a credential that did not exist.
• A tool returned a cached listing, and acting on it sent someone to fix something that was not broken.

The rule is to re-run the narrowest possible check on its own, with no pipe and no compound operator, before stating that something exists or does not. When a listing tool disagrees with what you expect, try the real operation before asking anyone to change configuration. And when a report turns out wrong, correct it in one plain sentence and keep going.

The same discipline applies to measurement. Query the real system you claim to report on, from a cold session, and save the evidence, as in measuring AI answers with Playwright.

What does a launch review look like?

We run it as a short, written pass before any production release.

1. Eval results with right, wrong, and no-answer counts and the sample size.
2. Logging confirmed by forcing an error and reading it.
3. Cost share of the model call in one transaction.
4. Infrastructure quotas listed, with the ones that can halt service marked.
5. Interface review result with screenshots.
6. Facts rechecked for any claim that will be reported to a customer.

For scoping the work before it gets this far, read how to scope an AI pilot.

Frequently asked questions

Want this built for your team? Book a free 30-minute call.