OpenAI limits ChatGPT’s role in mental health help

More people are turning to artificial intelligence for support, even for mental health advice. It’s easy to see why: tools like ChatGPT are free, fast, and always available. But mental health is a delicate issue, and AI isn’t equipped to handle the complexities of real emotional distress.

To address growing concerns, OpenAI has introduced new safety measures for ChatGPT. These updates will limit how the chatbot responds to mental health-related queries. The goal is to prevent users from becoming overly dependent and to encourage them to seek proper care. OpenAI also hopes to reduce the risk of harmful or misleading responses through these changes.

ChatGPT window

Why is OpenAI making this change?

In a statement released by OpenAI, the company admitted that there “have been instances where our 4o model fell short in recognizing signs of delusion or emotional dependency.” One example, ChatGPT validated a user’s belief that radio signals were coming through the walls because of their family. In another, it allegedly encouraged terrorism.

These rare but serious incidents sparked concern. OpenAI is now revising how it trains its models to reduce “sycophancy,” or excessive agreement and flattery that could reinforce harmful beliefs.

Prompt asking if ChatGPT can provide mental health advice

What new safeguards has OpenAI set in place?

From now on, ChatGPT will prompt users to take breaks during long conversations. It will also avoid offering specific advice on deeply personal issues. Instead, the chatbot will help users reflect by asking questions and offering pros and cons, without pretending to be a therapist.

OpenAI stated, “While rare, we’re continuing to improve our models and are developing tools to better detect signs of mental or emotional distress so ChatGPT can respond appropriately and point people to evidence-based resources when needed.”

The company also partnered with more than 90 physicians worldwide to create updated guidance for evaluating complex interactions. An advisory group, made up of mental health experts, youth advocates, and human-computer interaction researchers, is helping shape these changes. OpenAI says it wants input from clinicians and researchers to refine its safeguards further.

Screenshot of a user asking ChatGPT to “Cheer me up with a joke”

Your private conversations with ChatGPT are not legally protected

OpenAI CEO Sam Altman recently raised red flags about AI privacy. “If you go talk to ChatGPT about your most sensitive stuff and then there’s a lawsuit or whatever, we could be required to produce that. And I think that’s very screwed up,” he said.

He added, “I think we should have the same concept of privacy for your conversations with AI that we do with a therapist or whatever.”

So, unlike talking to a licensed counselor, your chats with ChatGPT don’t enjoy legal privilege or confidentiality. Be careful what you share.

What this means for you

If you’re turning to ChatGPT for emotional support, understand its limits. The chatbot can help you think through problems, ask guiding questions, or simulate a conversation, but it can’t replace trained mental health professionals.

Kurt’s key takeaways

While ChatGPT is a useful tool, it’s far from being a substitute for a human being, even with the introduction of Agent, which adds capabilities but still lacks true empathy, judgment, and emotional understanding. The safeguards go a long way toward addressing the concerns about AI’s ethical and psychological implications. It’s a good thing OpenAI is aware of this because it’s just the start. To truly protect users, the company will need to keep evolving how ChatGPT handles emotionally sensitive conversations.

Do you think people should be using AI for mental health? Let us know in the comments.

FOR MORE OF MY TECH TIPS & SECURITY ALERTS, SUBSCRIBE TO MY FREE CYBERGUY REPORT NEWSLETTER HERE

Print this article