Character.AI institutes new safety measures for AI chatbot conversations

Character AI
(Image credit: Character AI)

Character.AI has rolled out new safety features and policies for building and interacting with the AI-powered virtual personalities it hosts. The new measures aim to make the platform safer for all users, but particularly younger people. The update includes more control over how minors engage with the AI chatbot, more content moderation, and better detection of the AI discussing topics like self-harm.

Though not cited in the blog post about the update, Character AI linked to the announcement in a post on X expressing condolences to the family of a 14-year-old who spent months interacting with one of Character.AI's chatbots before taking his own life. His family has now filed a lawsuit against Character.AI for wrongful death, citing a lack of safeguards for the AI chatbots as a contributor to his suicide​.

AI chat guardrails

Character AI's post laid out several new safety features for the platform. For instance, if the model detects keywords related to suicide or self-harm, it will display a pop-up urging the user to the National Suicide Prevention Lifeline and related resources. The AI will also be better at spotting and removing inappropriate content in a conversation, with a particular sensitivity to when users are under 18.

Presumably, minors would already have restricted content in conversations, but Character.AI may have upped that sensitivity further. In cases where that might not be enough, entire chatbots have been removed.

"We conduct proactive detection and moderation of user-created Characters, including using industry-standard and custom blocklists that are regularly updated. We proactively, and in response to user reports, remove Characters that violate our Terms of Service," Character.AI explained in its post. "Users may notice that we’ve recently removed a group of Characters that have been flagged as violative, and these will be added to our custom blocklists moving forward."

Other new features are more about helping ground users. So, you'll see a notification when you have spent an hour on the platform asking if you want to keep going as a way of helping make sure you don't lose track of time. You'll also see more prominent disclaimers emphasizing that the AI is not a real person. There are already such disclaimers in the conversations, but Character.AI wants to make it impossible to ignore.

These safety features are the flipside of how Character.AI has made engaging with chatbots feel more like talking to a real person, including voices and the two-way voice conversations available with the Character Calls feature. Still, the company is likely keen to ensure its services are as safe as possible, and its moves could inform how others in the space shape their own AI chatbot characters.

You Might Also Like...

Eric Hal Schwartz
Contributor

Eric Hal Schwartz is a freelance writer for TechRadar with more than 15 years of experience covering the intersection of the world and technology. For the last five years, he served as head writer for Voicebot.ai and was on the leading edge of reporting on generative AI and large language models. He's since become an expert on the products of generative AI models, such as OpenAI’s ChatGPT, Anthropic’s Claude, Google Gemini, and every other synthetic media tool. His experience runs the gamut of media, including print, digital, broadcast, and live events. Now, he's continuing to tell the stories people want and need to hear about the rapidly evolving AI space and its impact on their lives. Eric is based in New York City.