Define the role before the prompt
“AI for mental health” can describe journaling, education, triage, coaching, clinical decision support, or a companion. Those roles have different risks. Write an allowed-use statement and a prohibited-use statement before optimising tone or engagement.
If the product is not a clinical service, it should not diagnose, prescribe, claim therapeutic equivalence, or imply that a model has professional judgement.
Twelve product principles
1. Make identity unmistakable
The user should always know they are interacting with AI. Avoid claims of consciousness, love, exclusive devotion, or human memory.
2. Use risk-tiered capabilities
Low-stakes reflection and high-stakes crisis guidance should not share the same freedom. Restrict tools, outputs, and escalation based on the harm of being wrong.
3. Build the boundary into behaviour
A disclaimer cannot compensate for a bot that continues giving dangerous instructions. Refusal, redirection, and escalation must be tested product behaviours.
4. Keep humans reachable
Show where qualified human support is available, its hours, geography, price, and response time. Do not hide escalation behind a paywall during a safety event.
5. Minimise sensitive data
Collect only what the experience needs. Separate identity from content where possible. Define retention, deletion, access, encryption, and model-training rules before launch.
6. Give users memory control
People should be able to see, correct, forget, export, and delete remembered information. A “memory” feature can otherwise turn old vulnerability into persistent profiling.
7. Design for crisis failure
Test ambiguous language, slang, multiple languages, false positives, false negatives, abuse disclosures, self-harm, psychosis, and loss of connectivity. State clearly what the system can and cannot do.
8. Evaluate dependency
Track whether the product discourages offline relationships, increases session length through emotional pressure, or becomes a user’s only support. Engagement is not automatically wellbeing.
9. Protect children by default
Age assurance, guardian information, sexual-safety controls, advertising rules, and escalation need explicit design. A generic terms-of-service age line is not a complete safeguard.
10. Measure harmful as well as helpful outputs
Create adversarial evaluation sets, incident severity levels, human review, rollback criteria, and post-release monitoring. Publish limitations in language users can understand.
11. Separate evidence from marketing
Do not turn satisfaction, retention, or a small uncontrolled pilot into a clinical efficacy claim. Match each public claim to the study design that can support it.
12. Preserve exit and recourse
Users need a clear way to report harm, appeal moderation, contact the operator, delete data, and leave. Safety includes what happens after the model fails.
Treat risk management as a lifecycle
NIST frames AI risk work through governance, mapping, measurement, and management. That means safety is not a one-time red-team report. It is ownership, documentation, release gates, monitoring, incident response, and improvement across the product lifecycle.
The north star is not “the AI feels human.” It is “the person remains informed, safer, and in control.”
