Patient Insists on AI-Generated Diagnosis, Refuses Medical Examination

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

In Taiwan, a patient brought 15 pages of ChatGPT-generated medical analysis to a clinic, demanding treatment based solely on the AI's recommendations and refusing standard clinical examination. The physician warned that overreliance on AI for diagnosis, without professional assessment, poses health risks and undermines medical expertise.[AI generated]

Why's our monitor labelling this an incident or hazard?

An AI system (ChatGPT) is explicitly involved as the patient used it to generate a medical diagnosis and demanded treatment based on it. The physician's account shows that the AI output was used in the patient's decision-making, but the physician did not follow the AI's advice blindly and insisted on clinical examination. No injury or harm has been reported, but the patient's overreliance on AI and refusal to allow proper medical assessment could plausibly lead to harm if acted upon. This fits the definition of an AI Hazard, where the AI system's use could plausibly lead to harm, but no incident has yet occurred. The event is not Complementary Information because it is not about a response or update to a prior incident, nor is it Beneficial Use since the AI is not used as a countermeasure but as a source of diagnosis by the patient. It is not Unrelated because AI is central to the event.[AI generated]
AI principles
SafetyAccountability

Industries
Healthcare, drugs, and biotechnology

Affected stakeholders
Consumers

Harm types
Physical (injury)

AI system task:
Content generationInteraction support/chatbots


Articles about this incident or hazard