An AI prototype has one important job: prove that a useful interaction might be possible.
Production software has a much harder job. It must keep creating value when users behave unexpectedly, dependencies fail, data is messy, costs vary and nobody from the original demonstration is standing nearby to explain what should happen.
The gap between those two states is where many AI products get stuck.
A demo proves the happy path
A compelling demonstration usually has selected inputs, a cooperative user and a builder who knows how to steer it. That is entirely reasonable during exploration. The problem begins when demonstration evidence is mistaken for production evidence.
Before launch, the product needs answers to questions the demo was never designed to address:
- How often is the output good enough for the intended decision?
- What happens when the model is slow, unavailable or changes behaviour?
- Can a malicious or confused user cause an unsafe action?
- How will the team investigate a poor result?
- What does each successful outcome cost?
- When must the system stop and involve a person?
1. Define quality before trying to improve it
“The answers look good” is not an evaluation strategy. Build a representative set of inputs and define what acceptable performance means for the actual workflow.
Some outputs can be checked automatically. Others need structured human review. The important point is to create a repeatable baseline so model, prompt and retrieval changes can be compared rather than admired.
Measure the outcome that matters. A fluent answer is not useful if it chooses the wrong appointment, cites an unsupported claim or creates work for the person who must correct it.
2. Put boundaries around agency
There is a material difference between an AI system that drafts a response and one that sends it, changes a record or triggers a financial action.
Define which tools the system may use, what data each tool can access, which actions require confirmation and how permissions are enforced outside the model. Treat model output as untrusted input to the surrounding application.
For higher-impact actions, prefer narrow tools, explicit validation and human approval. Convenience should not silently become authority.
3. Design failure as carefully as success
Models time out. APIs rate-limit. source data conflicts. Users ask for things outside the intended domain. Production design decides what happens next.
A useful fallback may be a constrained response, a retry with a different provider, a queue for human review, or a clean hand-off that preserves the context already collected. The user should not need to restart because one dependency had a bad minute.
4. Make the system observable
Traditional application logs are necessary but insufficient. An AI product may also need to record model and prompt versions, tool calls, retrieval sources, latency, token use, evaluation results and the reason a workflow was escalated.
That does not mean storing every sensitive input forever. Observability must respect data minimisation and retention requirements. Capture enough structured evidence to diagnose behaviour without creating a new privacy problem.
5. Control variable cost
An AI feature can appear cheap at demonstration scale and become commercially uncomfortable when usage grows. Track cost per completed outcome rather than only the monthly provider invoice.
Use model routing, caching, bounded context, sensible limits and asynchronous processing where they improve the economics without degrading the user experience. Put alerts around unexpected changes in volume and unit cost.
6. Build an ordinary secure application around the extraordinary model
AI-specific risks matter, but the familiar controls still matter too: authentication, authorisation, secrets management, dependency updates, encryption, backups and tested recovery.
The NIST Secure Software Development Framework is a useful baseline, while OWASP’s Agentic Top 10 helps teams reason about goal hijacking, tool misuse, identity and privilege abuse, supply-chain risk and unexpected code execution.
7. Strengthen the delivery system
AI accelerates the production of changes. That increases the value of automated tests, small batches, code review, progressive rollout and reliable rollback.
DORA’s research has found that AI can increase throughput while also increasing instability when the surrounding delivery system is weak. The answer is not to reject AI assistance. It is to strengthen the feedback loops that make speed safe.
Read DORA’s analysis of AI adoption and software delivery.
8. Decide who operates the product
Before launch, name the people responsible for releases, alerts, security updates, model changes, customer escalations and cost review. Document the important decisions and the recovery procedures.
A product is not production-ready merely because it has been deployed. It is ready when a team can operate it without depending on luck or the constant presence of its original builder.
A practical definition of production
I consider an AI product ready for production when its valuable workflow is measurable, its authority is bounded, its failures are controlled, its important behaviour is observable, its economics are understood, and somebody is accountable for operating it.
That work is less visible than the demo. It is also where customer trust is built.
Have a prototype that needs to become a product?
AlanOps can assess the current workflow, architecture, integrations and operational risks, then recommend the smallest credible path to production.