Crisis pop-ups differ across chatbots and one new law

What happened

Two companies and two state legislatures have separately published statements on what a chatbot should do when a user raises self-harm. Character.AI's Community Safety Updates post, dated 22 October 2024, describes 'a pop-up resource' triggered by 'certain phrases related to self-harm or suicide' that directs a user to the 'National Suicide Prevention Lifeline'. OpenAI's Helping people when they need it most, dated 26 August 2025, describes models 'trained' since 'early 2023' to refuse self-harm instructions and 'steer' users toward help. California's SB 243, chaptered 13 October 2025, requires an operator to maintain a 'protocol' referring a user to 'crisis service providers'. New York's General Business Law 1701 requires a notification naming the '9-8-8' hotline specifically.

What the documents show

Character.AI's and OpenAI's posts describe features already shipped by each company, in each company's own words, with no independent verification in these documents of how reliably either trigger fires. California's statute requires a protocol and public disclosure of it, but the statutory text reviewed does not mandate a specific hotline name. New York's statute is more prescriptive, naming '9-8-8' directly. Read together, the two company posts describe voluntary, self-reported features; the two statutes describe a legal floor, and only one of the two names a specific service.

The mechanism

Each of these responses works by pattern-matching a conversation against a list of trigger phrases, then substituting a scripted response, a referral message or a pop-up for whatever the underlying model would otherwise generate. This is a content-moderation layer bolted onto a conversational system, not a clinical safeguard built into how the model reasons; a phrase-matching trigger can both miss indirect expressions of risk and interrupt unrelated conversations that happen to contain a matched phrase.

What it leaves open

None of these four documents states how often the described triggers fire correctly, how often they miss, or what happens after a referral message is shown. That evaluation question, and the editorial reading required to weigh it, sits outside what a company blog post or a statute's text is designed to answer.

• Does the platform name a specific crisis service, or only describe a general referral?
• Is the response a scripted pop-up, a change in model behaviour, or both?
• Does a law require public disclosure of the protocol, or only the protocol's existence?

A pop-up naming a hotline is a real feature with a documented trigger and message, not a guarantee about what happens to the person who saw it after they closed the chat.