Launch readiness is not a feeling. It is a body of evidence.
Before AlanOps puts an AI product in front of customers, I want evidence that the valuable workflow works, the system fails safely, important behaviour can be explained, and somebody is ready to operate it.
The exact depth depends on the consequence of failure. A tool that drafts internal notes does not need the same controls as an agent that changes customer records. The categories, however, remain consistent.
1. Product acceptance
- The primary user and valuable workflow are explicit.
- Acceptance criteria describe outcomes, not only interface behaviour.
- The first-release boundary and exclusions are documented.
- Product analytics measure successful outcomes and important abandonment.
- Known limitations are communicated honestly to users and operators.
A technically impressive system that cannot demonstrate user value is not ready. It is an experiment that needs another product decision.
2. AI quality and evaluation
- A representative evaluation set covers normal, difficult and adversarial inputs.
- Quality criteria are defined for the actual workflow.
- Model, prompt, retrieval and tool changes can be compared against a baseline.
- Human review is used where quality cannot be judged reliably by automation.
- Regression thresholds can block a release.
Evaluation should reflect consequence. A harmless wording variation is not equivalent to an unsupported factual claim or an incorrect operational action.
3. Authority and human control
- Each tool has the narrowest permissions required.
- High-impact and irreversible actions require explicit approval.
- Inputs and model outputs are validated outside the model.
- Rate, spend and action limits are enforced by the application.
- A person can pause automation and recover the workflow.
OWASP’s Top 10 for Agentic Applications provides a useful threat vocabulary, including goal hijacking, tool misuse, identity and privilege abuse, supply-chain vulnerabilities and unexpected code execution.
4. Application security
- Authentication and authorisation are tested across tenant and role boundaries.
- Secrets are stored and rotated appropriately.
- Sensitive data is minimised, encrypted and retained intentionally.
- Dependencies, containers and infrastructure are scanned and maintained.
- Abuse cases and the most important threats have documented mitigations.
- Security-relevant actions produce a useful audit trail.
The NIST Secure Software Development Framework is a practical baseline for integrating these controls into delivery rather than treating security as a final gate.
5. Reliability and failure handling
- Timeouts, retries and circuit breakers are deliberate.
- Model and external-service failures have user-safe fallbacks.
- Duplicate requests do not create duplicate actions.
- Important asynchronous workflows can be reconciled.
- Backups exist and recovery has been tested.
- A release can be rolled back without improvisation.
We test degraded behaviour, not only the ideal path. Customers experience the fallback as part of the product.
6. Observability and economics
- Logs connect user-visible failures to the relevant workflow.
- Metrics cover latency, availability, exceptions and successful outcomes.
- Model, prompt and tool versions are traceable where necessary.
- Alerts identify conditions that require action rather than producing noise.
- Usage and cost are visible per customer or completed outcome.
- Unexpected volume and spend changes trigger controls.
An AI product cannot be managed from a provider’s monthly invoice. Unit economics and operational behaviour need to be observable while there is still time to respond.
7. Delivery and change control
- Changes pass automated checks and senior review.
- Production deployments are repeatable.
- Environment configuration is controlled and auditable.
- Releases can be staged, monitored and reversed.
- Database and data migrations have a recovery plan.
- Third-party and model changes are treated as product changes.
AI may accelerate code creation, which makes strong feedback loops more valuable rather than less. DORA’s research describes the technology as an amplifier of the delivery system around it.
Read the DORA State of AI-assisted Software Development report.
8. Ownership and operation
- A named person owns releases, alerts and customer-impacting incidents.
- Runbooks cover the failures most likely to require intervention.
- Architecture decisions and known risks are documented.
- The code, data and infrastructure ownership model is explicit.
- Support responsibilities and response expectations are agreed.
- A managed product has a credible handover and exit path.
The final launch question
The checklist ends with one question: would we be comfortable explaining this system’s decisions and failure controls to a customer after something went wrong?
If the answer is no, the missing evidence becomes launch work. If the answer is yes, it should be supported by tests, controls, documentation and named ownership—not confidence alone.
That is what vetted software means in practice.
Want an independent production-readiness review?
AlanOps can assess an existing prototype or planned build across product fit, architecture, security, reliability and operation, then turn the findings into a prioritised delivery plan.