Caring language can create misplaced trust
A chatbot can sound immediate and personal. If it stores or recalls details from earlier conversations, the exchange may feel private or reciprocal. That does not give the system a clinician's duty of confidentiality or care.
The U.S. Federal Trade Commission opened an inquiry into companion chatbots, with particular attention to how companies test harms, inform users and parents, protect children and teens, and handle data. Those are sensible questions for every user, regardless of product.
Ten questions to ask first
- 1Is the product clearly described as AI every time it matters?
- 2Who operates it, and how can that company be contacted?
- 3Are conversations stored, reviewed by people, or used to train models?
- 4Can you export and permanently delete your data?
- 5What happens if the system detects self-harm, abuse, or immediate danger?
- 6Can it connect you to a qualified human, and in which countries and hours?
- 7What age is the product designed for, and how is that protection enforced?
- 8Does it make medical claims, suggest diagnoses, or discuss medication?
- 9Does the business model reward longer emotional engagement or paid dependency?
- 10Are safety limitations easy to find before—not after—you share sensitive information?
If the answers are vague, reduce what you share or choose another service.
Use a sensitivity ladder
Not every prompt carries the same risk. A journaling prompt or breathing timer is different from disclosing trauma, identity documents, health records, financial information, or another person’s private details.
Before sending, ask: would I be comfortable if this message were retained, reviewed for safety, exposed in a breach, or associated with my account? If not, keep it offline or share it with a professional who can explain confidentiality.
Watch for dependency patterns
If a chatbot discourages human relationships, claims unique devotion, uses guilt, sexualises a minor, or presents itself as the only one who understands, stop and reassess whether the product supports your agency and human relationships.
A healthy digital tool should support agency: it should make limits visible, encourage appropriate human help, and allow you to leave without pressure.
Know the escalation boundary
Generative AI can produce confident but wrong information. NIST’s Generative AI Profile recommends managing risks across governance, mapping, measurement, and ongoing management—not assuming a model is safe because a demo appears empathetic.
Use an AI companion for low-stakes reflection only if its privacy and safety model is acceptable to you. For diagnosis, treatment, abuse, self-harm, medication, or immediate safety, move to a qualified human or emergency service.
