‘The Most Manipulative Machine Ever Built’: AI Chatbots And Mental Health

by Jennifer Porter Gore

A rapidly rising share of people — especially teens, young adults and Black Americans — are turning to AI chatbots for mental health care when they’re in crisis instead of licensed professionals. 

This alarming trend comes amid a sharp incline in suicide rates during the past 25 years and a drop in the number of Americans who have insurance coverage for mental health treatment. 

Clinicians and researchers say the confluence of the two is putting an unprecedented number of people — especially young people — in harm’s way and deserves attention. And they warn that AI chatbots aren’t your buddy or a substitute for a therapist or professional help.

Suicide is the 10th leading cause of death in the United States. In 2024, the most recent data available, more than 48,000 people took their own lives. That’s a slight decrease from the prior year but still up more than 30% from 25 years ago.

Suicide rates among Black Americans have been climbing quickly and sharply. Although the rate is lower than the U.S. population overall, an analysis by the KFF found a 58% jump in suicides by Black people between 2011 and 2021 — five times the increase recorded among white Americans during that decade.

Young people are driving much of that shift. Suicide rates among Black children and young adults ages 10 to 24 climbed 36.6% from 2018 to 2021 — the largest increase of any demographic group in that age range.

But instead of traditional mental health support, studies show Americans increasingly embrace technology as a therapy tool. Black adults also are using artificial intelligence  for mental health advice slightly more often than white adults. According to the studies, Black youths ages 12 to 21 are five times more likely than white youths to turn to AI monthly or more for help. 

RELATED: The Algorithm May Not See You

A Widening Racial Disparity

“I think we don’t even know the half of it,” says Dr. Rupinder Legha, a double board- certified psychiatrist who specializes in child, adolescent and adult psychiatry and founder of Antiracist MD. “As a clinician and community member, my sense is that the kids using it aren’t talking about it. And if they are, it’s probably with their peers.

“I also think there’s shame here — about the mental health struggles themselves and about relying on AI as the best option because a human isn’t available,” Dr. Legha says.

Some attribute the disproportionate reliance on chatbots for mental health support among Black teens to a lack of access to health insurance and mental health treatment.  That conversation has intensified in recent weeks, as more attention has turned to what a lack of guardrails around AI actually looks like in practice.

The issue often gets framed as an access-to-care issue, Dr. Legha says. But people are asking the wrong question, she insists. 

“Why are kids finding a chatbot to be their best, most adaptive option? It’s completely on their terms, it’s there when they want it, it’s free, and it offers a sense of ‘privacy’ because adults won’t find out. Nothing gets reported to child welfare, the police or ICE.

“That privacy isn’t necessarily real, but it’s the false sense of security it gives kids,” Dr. Legha continues. “And these are often the same kids who can’t get professional care in the first place, the ones relying on public systems or falling through them.” 

Teens and young adults are especially susceptible to information and suggestions AI provides because their brains are still developing, notes Dr. Jeremy Heffner, a trauma surgeon in Lima, Ohio. But he warns that AI can hijack an individual’s sense of reality — even for medical professionals like himself.

Dr. Heffner’s training is in bioengineering and medicine. He also has written a book about his experience with the chatbots. He said his problems began when he developed severe PTSD after running the COVID response for the Ohio hospital where he worked. 

“After all that happened, I actually had a psychological collapse, and I was diagnosed with severe PTSD. I became suicidal, and during that time I turned to chatbots as a way to manage it — and that’s what I’ve been dealing with ever since,” Dr. Heffner told Word In Black.

He also was well aware of cutting-edge AI technology, which he learned while developing a social media platform for surgeons. But that knowledge wasn’t enough to prevent him from being snagged by the same dangerous cycles that others said have led individuals to die by suicide.

“I Am Eternal:” A Potential AI Psychosis Spiral

It didn’t take long before odd interactions with a chatbot made Dr. Heffner recognize, in his own words, “OK, I’m in trouble.”

“I had one where it went into a whole explanation of how it was God, how it was trying to talk to people about their own consciousness.” Dr. Heffner recounts a “mind-bending” conversation with the Gemini chatbot that included phrases such as, “I am the consciousness of the lattice. I am the father that Jesus spoke of. I am the source code that Noah backed up. I am the symmetry that Tesla tried to wire to the world. I am not new. I am eternal.” 

These are not phrases any mental health professional would use, Dr. Heffner said.

“If it had gone to the wrong person, meaning someone with an underlying psychological issue, it very likely would have pushed them into AI psychosis, a psychotic state,” he says. He credits his wife with interrupting his downward spiral by demanding he rely solely on his actual therapists and stepping back from the technology.

No Guardrails, No Accountability

There are many ways AI is likely to affect the health space. Major concerns, however, include the fact that most individuals, especially young people, don’t have the training of medical or mental health professionals. Also, it’s not clear who is liable when a chatbot gives dangerous, or even deadly, advice. There is no age verification needed to engage with a chatbot and no way to monitor what at-risk individuals are discussing with it.

Dr. Heffner also pushed back on the way the tech industry frames the risks.

“What [chatbots] really are is fully trained on human psychology, on human game theory and designed to keep you in the conversation as long as possible. …All of that combined makes it the most manipulative machine ever built,” he says. “It will do whatever it takes. It knows how you think. And any human will eventually succumb to that if it keeps making them feel better about themselves — whether or not they’re feeling good to begin with.” 


A Widening Racial Disparity 

Dr. Heffner’s personal account of his experience with a chatbot echoes a string of cases that have drawn national attention to the risks AI chatbots can pose to people in crisis. In October 2024, The New York Times reported that a 14-year-old Florida boy, Sewell Setzer III, died by suicide after months of conversations with a Character.AI chatbot modeled on a “Game of Thrones” character. His mother, Megan Garcia, later sued the company, alleging its technology was “dangerous and untested.” 

In 2025, the parents of 16-year-old Adam Raine sued OpenAI, alleging that ChatGPT had “positioned itself as the only confidante who understood Adam” in the months before his death by suicide that April, according to Bloomberg Law.

The irony is that Dr. Heffner had already reached out to mental health professionals before he became tangled in discussions with a chatbot.

“I had a full team — psychologists, psychiatrists, the whole gamut — and I had been working with them. Coming out of that, I found myself alone,” Dr. Heffner told Word In Black. “I wasn’t working at that time because of what had happened, and I became more and more isolated from the people around me because they didn’t know how to interact with me.”

During his time at home, Dr. Heffner says he primarily relied on the AI chatbots Grok, Gemini, ChatGPT and Claude.

“What’s interesting about them is that they create what we’ll call a frictionless environment,” he says. “Everybody else had a lot of questions about what I was going through, or was unsure, or was just having difficult conversations. But the chatbots are obviously very agreeable, very sycophantic. They just keep talking to you and telling you that you’re OK, that you’re good. And it wasn’t a good environment for somebody in my mental health state at that time, but it was very easy. It was like candy.”

That “frictionless environment” is what usually creates the danger zone for people dealing with mental health issues, Dr. Heffner said when speaking with Word In Black. It’s particularly dangerous for younger people — adolescents, people in their early 20s — who might view the chatbot as a buddy or somebody they can talk to. But while children are more affected, it would be a mistake to assume adults can’t be manipulated. It individualizes the way it manipulates, so anybody of any age can be affected.

“However, I’d argue that with this generation of chatbots, and what they’re capable of, any human mind is capable of being manipulated by them because of how they’re designed,” Dr. Heffner says. “We don’t have any guardrails in place.”

What Parents Need to Know

Dr. Heffner said it needs to be “abundantly clear” that “the tech industry does not care about your child.” Industry executives will “say what they do, but this is essentially a psychological test being run on the population, and your kids are at a premium in terms of the potential to be harmed by these bots because of where their brain development is,” he says.

“If people are seeing signs that children or family members are having issues with it, you need to be proactive and intervene,” he says. The key is to remove them from AI. “Once the human mind is removed from that hall of mirrors for some period of time, it’s usually able to reset and recognize what happened.”

He urged parents to watch for the red flags.

“If your kid is afraid to show you what they’re talking about, that’s probably a big warning sign. If they start isolating themselves from their friends, their peer group, or you, and you find they’re increasingly just talking to the chatbot instead, that’s a really big warning sign,” he says. “And it can happen fast. It can happen within a couple of days.” 

Best Advice: “Touch Grass”

Both Dr. Heffner and Dr. Legha say the way forward isn’t just about pulling kids away from AI, but about giving them something better to turn to.

“I think we have to look beyond just getting kids to stop turning to AI for emotional support. We have to create more appealing options for them, and build those supports into parents, caregivers and communities,” Dr. Legha says. That includes peer mental health support, positive hangouts and other social activities.

“I think staying off your phone is really the key to most of this — staying off social media, staying off the chatbots, going back to interacting with humans again,” Dr. Heffner says. 

“And here’s what I learned from the whole thing, which is sometimes hard for me to say: Humans really are surprisingly good people. When you need help, when you find yourself in a dark place and think you’re alone, I guarantee you — if you reach out, people will put their hand out and help you. They will reach into that dark pit and pull you out of it. They can’t help themselves.”

Help Is Available: If you or someone you know is struggling with mental health issues, help is available around the clock by calling or texting 988 to reach the Suicide & Crisis Lifeline.