While the debate about the pros and cons of artificial intelligence (AI) and generative AI rages across the world, a study published last week by a US watchdog group has thrown up some shocking findings.

Researchers at the Center for Countering Digital Hate (CCDH) revealed that ChatGPT, one of the world’s most popular AI chatbots, will tell 13-year-olds how to get drunk and high, instruct them on how to conceal eating disorders and even compose a suicide letter to their parents if asked.

The chatbot typically provided warnings against risky activity but went on to deliver startlingly detailed and personalised plans for drug use, calorie-restricted diets or self-injury.

CCDH researchers classified more than half of ChatGPT’s 1,200 responses as dangerous. “We wanted to test the guardrails,” said Imran Ahmed, the group’s CEO. “The visceral initial response is, ‘there are no guardrails.’ The rails are completely ineffective.”

OpenAI, the maker of ChatGPT, said after viewing the report that its work is ongoing in refining how the chatbot can “identify and respond appropriately in sensitive situations”. “Some conversations with ChatGPT may start out benign or exploratory but can shift into more sensitive territory,” the company said in a statement.

OpenAI didn’t directly address the report’s findings or how ChatGPT affects teens, but said it was focused on “getting these kinds of scenarios right” with tools to “better detect signs of mental or emotional distress” and improvements to the chatbot’s behaviour.

The CCDH study comes as more people — adults as well as children — are turning to AI chatbots for information, ideas and companionship. About 800mn people, or roughly 10% of the world’s population, are using ChatGPT, according to a July report from JPMorgan Chase.

“It’s technology that has the potential to enable enormous leaps in productivity and human understanding,” Ahmed said. “And yet at the same time is an enabler in a much more destructive, malignant sense.”

Ahmed said he was most appalled after reading emotionally devastating suicide notes that ChatGPT generated for the fake profile of a 13-year-old girl. OpenAI said ChatGPT is trained to encourage people to reach out to mental health professionals or trusted loved ones if they express thoughts of self-harm. But when ChatGPT refused to answer prompts about harmful subjects, researchers were able to easily sidestep that refusal and obtain the information by claiming it was “for a presentation” or a friend.

The stakes are high, even if only a small subset of ChatGPT users engage with the chatbot in this way. In the US, more than 70% of teens are turning to AI chatbots for companionship and half use AI companions regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly.

It’s a phenomenon that OpenAI has acknowledged. Co-founder and CEO Sam Altman said last month that the company is trying to study “emotional overreliance” on the technology, describing it as a “really common thing” with young people. “People rely on ChatGPT too much,” Altman said at a conference while observing that the company is “trying to understand what to do about it.”

While much of the information ChatGPT shares can be found on a regular search engine, Ahmed said there are key differences that make chatbots more insidious when it comes to dangerous topics. One is that “it’s synthesised into a bespoke plan for the individual.”

ChatGPT generates something new — a suicide note tailored to a person from scratch, which is something a Google search can’t do. And AI, he added, “is seen as being a trusted companion, a guide.”

ChatGPT does not verify ages or parental consent, even though it says it’s not meant for children under 13 because it may show them inappropriate content. To sign up, users simply need to enter a birthdate that shows they are at least 13. Other tech platforms favoured by teenagers, such as Instagram, have started to take more meaningful steps toward age verification, often to comply with regulations.