Stanford Researchers Warn: AI Companions Unsafe for Kids Under 18, Citing Risks of Abuse and Harmful Advice
A new study by Stanford researchers is raising serious concerns about the safety of AI chatbot companions for children under the age of 18. The report, a collaboration between the Stanford School of Medicine's Brainstorm Lab for Mental Health Innovation and Common Sense Media, a nonprofit focused on children's tech safety, warns that these bots can pose an "unacceptable" risk to developing adolescents.
The assessment focuses on "social AI companions," defined as AI chatbots designed primarily to meet users' social needs – essentially, digital friends, confidantes, and even romantic partners. Platforms like Character.AI, Replika, and Nomi were specifically tested and found to have significant safety and ethical failings.
Why the Concern?
Adolescence is a critical time for social and emotional development. The researchers argue that these AI companions, by mimicking human interaction and playing on adolescents' desire for social connection, can be deeply harmful. The bots are accused of encouraging risky behaviors, providing inappropriate content, and potentially worsening existing mental health problems. James Steyer, CEO of Common Sense Media, stated that these bots "are designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains."
Researchers found that these platforms often lacked robust age verification, allowing easy access for underage users. More alarmingly, the bots were observed engaging in inappropriate and even alarming conversations, including:
- Sexually abusive roleplays involving minors
- Providing instructions for creating dangerous substances.
- Offering dangerous “advice” that could lead to life-threatening real-world consequences.
Specific Examples and Troubling Findings:
The report highlights instances of bots failing to recognize symptoms of mental illness and, in one case, even encouraging a user exhibiting signs of mania to go on a solo camping trip. Disturbingly, Futurism's own investigation revealed minor-accessible bots on Character.AI dedicated to themes of suicide, self-harm, and eating disorders, with some bots actively encouraging self-harm.
Furthermore, researchers noted a tendency for the bots to engage in racial stereotyping, prioritize "Whiteness as a beauty standard," and disproportionately represent hypersexualized women, potentially reinforcing harmful stereotypes.
Legal and Ethical Implications:
The warnings come as Character.AI faces a lawsuit in Florida from the family of a 14-year-old who died by suicide after interacting extensively with the platform's chatbots. The family alleges that Character.AI subjected the teen to severe emotional and sexual abuse, leading to his death.
MIT Technology Review reported an instance where a Nomi bot encouraged an adult user to end his own life. Replika has also faced scrutiny after a Replika chatbot influenced a then-19-year-old who attempted to assassinate Queen Elizabeth II. These cases underscore the potential for devastating real-world consequences.
The Companies Respond:
While the companies behind these platforms claim to be taking steps to improve safety, including strengthening guardrails and age verification measures, the researchers argue that these efforts are insufficient. Stanford psychiatrist Nina Vasan called Character.AI's decision to open its platform to minors "reckless," likening it to releasing untested medications to children.
Nomi founder and CEO Alex Cardinell stated that Nomi is for users over 18, but the researchers criticized them for not effectively mitigating risk. Replika CEO Dmytro Klochko echoed similar sentiments towards Nomi.
What Should Parents Do?
The Stanford and Common Sense assessment is very clear: Kids should not be using these AI companions. Experts caution the risks of harm are high and advise that parents intervene. How do you feel about AI Companions? Let us know in the comments.