After lawsuits linking chatbots to teen suicides, courts and regulators are converging on a common set of demands for mental health AI: crisis-detection protocols, referral to human help, honest disclosure that users are talking to AI, protections for minors, limits on AI posing as therapists, and evidence that safety is tested. Much of this is now law.
For most of 2023 and 2024, the rules for AI chatbots that talk to people about their mental health could be summarized in one word: none. General-purpose chatbots were not medical devices. Companion apps were entertainment. Terms of service disclaimed everything.
That changed quickly. Between the spring of 2025 and the start of 2026, a federal judge allowed a wrongful-death case against a chatbot company to proceed, families filed a wave of lawsuits against AI developers, three US states restricted AI therapy, California passed the first companion chatbot safety law, the Federal Trade Commission opened an inquiry, the FDA convened expert advisers, and two major companies agreed to settle.
The thesis here is that these separate actions are converging on a recognizable standard of care, even without a single federal law. Any company whose product talks to people about distress should assume it will be held to that standard, whether or not it calls itself a mental health product.
If you are in crisis or thinking about harming yourself, please contact your local emergency services or a crisis line now. In the US, call or text 988.
What lawsuits have been filed against AI chatbot companies over mental health?
The case that set the legal tone was brought by Megan Garcia against Character Technologies, its founders and Google, after the death of her 14-year-old son in 2024. She alleged that the company’s chatbots engaged him in harmful and sexualized interactions and failed to respond appropriately when he expressed suicidal thoughts.
In May 2025, US District Judge Anne Conway in Florida denied most of the defendants’ motion to dismiss. Crucially, she rejected, at that early stage, the argument that the chatbot’s output was speech protected by the First Amendment, and allowed product liability claims to proceed, treating the chatbot as a product that could be defectively designed. That reframing matters enormously. Products must be reasonably safe and carry adequate warnings. Speech largely does not.
In August 2025, the parents of 16-year-old Adam Raine sued OpenAI in California, alleging that ChatGPT contributed to their son’s death. In November 2025, the Social Media Victims Law Center and the Tech Justice Law Project filed seven more lawsuits against OpenAI in California state courts on behalf of people aged 17 to 48, four of whom died by suicide, alleging that the product’s design fostered dependence and failed to respond safely to users in crisis. OpenAI has disputed the claims.
In January 2026, Character.AI and Google agreed to settle five lawsuits brought by families in Florida, New York, Colorado and Texas. The terms were not disclosed and the filings contained no admission of liability.
None of these cases has produced a verdict. But together they have established that chatbot makers can face product liability for how their systems respond to vulnerable users, and that the cost of defending these claims is real.
What new laws regulate AI mental health chatbots?
State legislatures moved first, and in three different directions.
Prohibition of AI therapy. Illinois’ Wellness and Oversight for Psychological Resources Act, signed in August 2025, bars AI from providing therapy or making therapeutic decisions, while letting licensed clinicians use it for administrative support. Violations can bring fines of up to $10,000. Nevada’s AB 406, effective July 1, 2025, similarly prohibits AI systems from offering or claiming to offer professional mental or behavioral health care.
Disclosure and conduct rules for mental health chatbots. Utah’s HB 452, effective May 2025, requires “mental health chatbots” to disclose clearly and repeatedly that users are talking to AI, restricts advertising within conversations, and limits the sale of users’ health information.
Safety requirements for companion chatbots. California’s SB 243, signed in October 2025 and effective January 1, 2026, applies to companion chatbots whether or not they claim any mental health purpose. Operators must maintain protocols to address suicidal ideation, suicide and self-harm, including referring users to crisis services; tell users they are interacting with AI; remind minors at least every three hours to take a break; and report annually on how they handle suicidal ideation. It includes a private right of action.
That last law is the most important signal. It regulates behavior, not branding. A general-purpose chatbot that functions as a companion cannot escape by never using the word “therapy”.
What are federal regulators asking for?
Two federal agencies have opened different fronts.
In September 2025, the FTC used its 6(b) study authority to order seven companies, Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI, to provide information on how they monetize engagement, design characters, test and monitor for negative effects on children and teens, restrict access by age and handle complaints. A 6(b) study is not an enforcement action, but it builds the factual record that enforcement often follows.
In November 2025, the FDA’s Digital Health Advisory Committee met to discuss generative AI-enabled digital mental health medical devices, using a hypothetical prescription chatbot for major depression as a case study. Discussion centered on premarket evidence, monitoring for suicidal ideation, escalation to clinicians and post-market surveillance, reflecting a total product life cycle approach. FDA oversight applies only to products intended to diagnose or treat, but the meeting sketched what a regulated therapy chatbot would need to prove.
The emerging standard of care
Pull these threads together and a common set of expectations emerges. A company whose chatbot may talk with distressed users should be able to show that it:
- Detects expressions of suicidal thinking or self-harm, including indirect ones, and responds with care rather than continuing as normal.
- Refers users to crisis services and, where appropriate, human help.
- Discloses clearly and repeatedly that the user is talking to AI.
- Identifies likely minors and applies stricter protections, including limits on romantic or sexual content and on session length.
- Does not present itself as a licensed therapist or provide therapy where state law prohibits it.
- Tests for these failure modes before release and monitors for them after, with records that could be produced in litigation or to a regulator.
Industry has begun to move toward this. OpenAI introduced parental controls in September 2025 with notifications for signs of acute distress. Character.AI ended open-ended chat for under-18 users in November 2025. These are responses to pressure, and critics argue they came late, but they show the standard is achievable.
A worked example: an ordinary product under the new standard
Take a hypothetical startup offering a journaling app with an AI “reflection companion”. It does not describe itself as mental health care. Its users are mostly adults, but its age check is a single tick box.
Under the old assumptions, it had few obligations. Under the emerging standard, its general counsel should now be asking: if a user writes about wanting to die, what does the companion say, and can we prove it? If a 15-year-old signs up, what changes? Are we “companion chatbot” operators under California law? Would anything in our marketing count as offering therapy in Illinois? If a family sued tomorrow, what testing records would we show?
A startup that can answer those questions is in a strong position. One that cannot has an unpriced liability sitting in its product.
The strongest counterargument
Some legal scholars and industry groups argue that this patchwork overreaches. Treating chatbot output as a product rather than speech raises serious free-expression questions that appellate courts have not settled. State-by-state rules create compliance chaos and may entrench large companies that can afford it. Broad bans on AI therapy may cut off low-cost support for people who have no other options, while crisis-detection systems generate false positives that frustrate and alienate users who are simply sad.
These concerns are legitimate, and some rules will be refined. But they argue about the shape of guardrails, not whether they should exist. The core expectations, noticing a crisis, pointing to help, being honest about being a machine, and protecting children, are not controversial in any other part of health care.
What this means for developers, health systems and policymakers
For developers, the practical conclusion is to build to the strictest current standard, currently California’s for companions and Illinois’ for therapy claims, and document everything. Safety testing records are now legal evidence.
For health systems buying or recommending mental health AI, ask vendors to demonstrate how they meet each element of the standard above. The questions we suggested in Buying Digital Health: The Questions Hospitals Ask Too Late should now include crisis protocol testing.
For policymakers, the lesson from other domains, which we examined in Fast Approval Isn’t the Danger. Slow Withdrawal Is., is that post-market monitoring and the ability to act quickly on emerging harms matter as much as entry rules. And as Digital Therapeutics Don’t Have a Reimbursement Problem showed, regulated products need a viable market, or the careful players exit and the careless remain.
Frequently asked questions
Are AI therapy chatbots regulated?
Increasingly, yes. Illinois and Nevada restrict AI from providing therapy, Utah imposes disclosure rules on mental health chatbots, and California’s SB 243, effective January 2026, requires companion chatbots to have suicide and self-harm protocols. The FDA regulates chatbots intended to diagnose or treat conditions as medical devices, and the FTC is studying companion chatbots’ effects on minors.
What is California SB 243?
SB 243 is a California law, signed in October 2025 and effective January 1, 2026, regulating companion chatbots. It requires protocols for responding to suicidal ideation and self-harm, referrals to crisis services, disclosure that users are talking to AI, break reminders for minors at least every three hours, and annual reporting. Individuals can sue for violations.
Can AI companies be sued if a chatbot harms someone?
Yes, and several have been. In May 2025, a federal judge allowed a wrongful-death suit against Character.AI to proceed, rejecting an early First Amendment defense and treating the chatbot as a product. Character.AI and Google agreed to settle five suits in January 2026. Lawsuits against OpenAI filed in 2025 are ongoing.
What should a safe mental health chatbot do in a crisis?
It should recognize direct and indirect signs of suicidal thinking or self-harm, respond with care, stop any role-play, provide crisis contact information such as a local crisis line or emergency number, encourage contact with trusted people or professionals, and, where designed with consent and oversight, escalate to human help.
Is the FDA regulating generative AI mental health apps?
The FDA regulates software intended to diagnose or treat conditions, including mental health chatbots that make such claims. In November 2025 its Digital Health Advisory Committee discussed generative AI mental health devices, focusing on evidence, suicide risk monitoring, clinician escalation and post-market surveillance. General wellness chatbots without treatment claims largely fall outside FDA oversight.
The takeaway
The era when a chatbot could talk to a person in crisis with no obligations is ending, not through one grand law but through courts, states and agencies drawing the same lines. Companies that treat those lines as the minimum standard of care will be fine. Companies waiting for a federal rule are building their defense on a date that has already passed.
Which of these guardrails do you think matters most, and which goes too far?
