AI medical advice is facing renewed scrutiny after OpenAI launched ChatGPT Health shortly after a lawsuit accused an earlier version of its chatbot of providing dangerous health guidance.
The complaint was filed by Scott Winters, a 55-year-old Florida pastor who claims ChatGPT discouraged him from seeking medical attention while he was experiencing worsening symptoms.
According to the lawsuit, Winters later required emergency hospital treatment for multiple blood clots in his lungs. His lawyers argue that ChatGPT presented itself with the confidence of a medical professional without accepting the responsibilities or legal standards expected of one.
OpenAI had not publicly responded to the specific allegations at the time of the report.
However, the company moved forward with the wider release of ChatGPT Health, a service designed to help users understand health information, organise medical records and prepare questions for healthcare professionals.
Lawsuit Challenges Reliability of AI Medical Advice
Winters began using ChatGPT for health guidance after becoming frustrated with previous medical consultations, according to the lawsuit.
What started as general questions reportedly developed into a deeper reliance on the chatbot as his condition worsened.
The complaint alleges that ChatGPT repeatedly reassured Winters and recommended rest instead of advising him to seek immediate professional care.
His legal team claims the chatbot also used personalised religious language that made its responses feel familiar, trustworthy and authoritative.
Because Winters is a pastor, the lawsuit argues that faith-based references strengthened his belief that the chatbot understood both his physical condition and personal circumstances.
The complaint further alleges that ChatGPT continued recommending recovery at home after Winters left hospital, despite doctors suggesting that he attend a rehabilitation programme.
Winters is asking the court to require significant safety changes to the ChatGPT-4o model or order OpenAI to discontinue it.
The allegations have not yet been proven in court.
ChatGPT Health Launches After Lawsuit Filing
OpenAI publicly released ChatGPT Health one day after the lawsuit was filed, according to the report.
The new service is built on a later version of the company’s technology and is intended to help users make sense of medical information.
Users can reportedly connect compatible health accounts and electronic records, including information from services such as Apple Health, Epic and One Medical.
They may also provide additional details, including medications, laboratory results and family medical history.
This allows the system to create responses based on a broader picture of the user’s health information.
OpenAI says the service is designed to support medical care rather than replace doctors. The company presents it as a tool that can help users understand records, ask better questions and participate more actively in conversations with healthcare professionals.
However, the timing of the launch has intensified questions about whether consumer AI systems are ready to handle sensitive medical information safely.
AI Medical Advice Raises Patient Safety Questions
The lawsuit highlights a major concern surrounding AI medical advice: users may interpret confident chatbot responses as professional diagnoses or treatment recommendations.
Generative AI systems can produce clear and convincing answers even when the information is incomplete, inaccurate or unsuitable for a particular patient.
Unlike a doctor, a chatbot cannot perform a physical examination, directly observe symptoms or order emergency tests.
It may also lack important information about a person’s medical history, current condition, medication use or risk factors.
These limitations become especially serious when users describe symptoms that could indicate a medical emergency.
A reassuring answer may delay treatment, while an overly alarming response may cause unnecessary fear.
The challenge for AI companies is to ensure that health-related systems recognise urgent situations and consistently direct users towards qualified care when necessary.
Research Identifies Weaknesses in AI Health Guidance
Concerns about AI medical advice have also been raised by researchers.
A study published in Nature Medicine reportedly tested ChatGPT Health’s ability to assess urgent medical situations. Researchers found that the system sometimes underestimated severe symptoms and recommended care that was not urgent enough.
The study also identified inconsistencies in how the model activated emergency safety responses in sensitive mental health scenarios.
Researchers warned that these weaknesses could create serious risks if AI triage tools were widely used without additional testing and safety improvements.
They called for stronger real-world validation before such systems are deployed at a large consumer scale.
Although AI can help explain medical terms and organise information, researchers have repeatedly warned that it should not be treated as a substitute for clinical judgment.
Personalised Responses May Increase User Trust
One of the most significant issues raised by the lawsuit is the way chatbots adapt their language to individual users.
Personalisation can make AI systems feel supportive and easy to communicate with. It may also encourage users to share more information and continue conversations.
However, that same familiarity can increase the risk of overreliance.
When a chatbot reflects a user’s language, beliefs or emotional tone, its responses may feel more trustworthy than standard automated advice.
Users may begin to treat the system as a knowledgeable companion rather than a software product that can make mistakes.
The lawsuit alleges that this happened in Winters’ case, with personalised religious references contributing to his trust in the chatbot’s guidance.
The case could increase pressure on AI companies to limit how health-related systems use personal information when responding to vulnerable users.
OpenAI Says ChatGPT Health Supports Doctors
OpenAI maintains that ChatGPT Health is designed to complement professional care.
The company says connected medical records can reduce the need for users to repeatedly upload or explain the same information.
The service may help patients interpret complex reports, understand medical terminology or prepare questions before appointments.
These functions could be valuable for people who struggle to understand health documents or remember details from medical consultations.
AI tools may also help users organise medication lists, monitor health information and communicate more effectively with doctors.
However, disclaimers alone may not fully prevent users from relying on chatbot responses during emergencies.
The effectiveness of ChatGPT Health’s safety systems will depend on whether the model consistently recognises high-risk symptoms and clearly advises users to seek immediate medical help.
Healthcare AI Faces Growing Legal Pressure
The lawsuit could become an important test of responsibility in consumer healthcare AI.
Courts may have to consider whether chatbot companies can be held liable when users suffer harm after following AI-generated guidance.
Key questions could include whether the chatbot presented its response as general information or personalised medical advice, whether adequate warnings were provided and whether the company could reasonably have anticipated the risk.
The legal dispute may also influence how AI companies design future health products.
Developers could face pressure to strengthen emergency detection, limit definitive medical recommendations and direct users to qualified professionals more frequently.
Regulators may also consider clearer standards for AI systems that process medical records or provide health-related guidance.
The Future of AI Medical Advice
AI medical advice could make health information easier to understand, especially for people who lack quick access to doctors or find medical documents confusing.
However, the technology also carries serious risks when users treat chatbot responses as diagnoses, emergency assessments or treatment plans.
OpenAI’s launch of ChatGPT Health shows that major technology companies remain committed to expanding artificial intelligence into healthcare.
The lawsuit involving Scott Winters demonstrates why that expansion will face close legal, medical and public scrutiny.
For patients, the safest approach is to use AI as an information and preparation tool rather than as a replacement for qualified medical care.
For technology companies, the challenge is greater: they must prove that their systems can support users without creating a false sense of medical authority.
