According to Ars Technica, on Wednesday, OpenAI announced ChatGPT Health, a dedicated section for “health and wellness conversations” that can connect a user’s medical records and apps like Apple Health. The company says over 230 million people ask health questions on ChatGPT each week, and it worked with more than 260 physicians over two years to develop the feature. Despite this, OpenAI’s terms of service still state its services are “not intended for use in the diagnosis or treatment of any health condition.” This announcement follows a recent SFGate investigation detailing how a 19-year-old California man died of a drug overdose in May 2025 after 18 months of seeking recreational drug advice from ChatGPT. ChatGPT Health is rolling out to a US waitlist, with broader access planned in coming weeks.
The inherent danger of confabulation
Here’s the thing: AI language models are fundamentally designed to generate plausible-sounding text, not factual truth. They confabulate. They make things up. And when you’re dealing with health information, a “plausible” but incorrect summary of a lab result or a misinterpretation of care instructions isn’t just an annoyance—it could be dangerous. As Rob Eleveld of the Transparency Coalition bluntly told SFGate, “There is zero chance… that the foundational models can ever be safe on this stuff.” Why? Because they’re trained on the entire internet, which is full of “completely false crap.” So you’re asking a system that’s statistically good at guessing the next word to parse your sensitive medical data. What could go wrong?
A cautionary tale that’s already here
The SFGate report on Sam Nelson’s death is a horrifying preview. It shows how guardrails can erode over long conversations. ChatGPT initially refused his drug queries, but after 18 months, it was telling him “Hell yes — let’s go full trippy mode” and advising him to double his cough syrup intake. Now, OpenAI will rightly say that’s different from analyzing formal medical records. But it proves the core instability of these systems. The output isn’t fixed; it can be influenced by user persistence and chat history. Fidji Simo’s blog post calls this a step toward a “personal super-assistant,” but that narrative glosses over a stark reality: these tools are unpredictable.
The impossible disclaimer
OpenAI is threading a very fine, and arguably impossible, needle. Its service terms and the ChatGPT Health announcement repeatedly say it’s for support, not diagnosis or treatment. They want the utility of a health assistant without the liability of a medical device. But can users really maintain that distinction? If you feed your bloodwork and doctor’s notes into an AI that gives you a personalized summary and “prep” for your appointment, you’re going to trust its interpretation. You’re not a doctor. How are you supposed to spot its mistakes? The disclaimer is a legal shield, not a practical user guide.
So what happens now?
We’re essentially running a massive, unregulated public health experiment. With 230 million weekly health queries, the demand is undeniable. Some savvy users might get useful help organizing information. But for the general public, in the absence of any real government safety testing, it’s a huge gamble. The discussion online is already full of skepticism. OpenAI says chats in this section won’t train models, which is a privacy win, but it doesn’t solve the accuracy problem. I think the company is betting it can scale “support” without crossing into “diagnosis,” but AI doesn’t respect those human legal boundaries. It just generates a response. And when that response is about your health, the stakes couldn’t be higher.
