AI Chatbots Found to Give Inaccurate and Risky Health Advice

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A large-scale study led by Oxford University found that AI chatbots, including those based on large language models, frequently provide incorrect or inconsistent medical advice, sometimes failing to recognize urgent health needs. The research warns that relying on these chatbots for health guidance can be dangerous and is not superior to traditional sources.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly involves AI systems (large language model chatbots) used for health advice. It documents that these AI systems provide bad advice and wrong diagnoses, which can directly harm users' health. The harm is related to injury or harm to persons, fulfilling the definition of an AI Incident. The study's findings indicate that the AI systems' outputs have led or could lead to health harm, not just a potential hazard but an actual realized risk. Hence, the event is classified as an AI Incident rather than a hazard or complementary information.[AI generated]
AI principles
SafetyHuman wellbeing

Industries
Healthcare, drugs, and biotechnology

Affected stakeholders
Consumers

Harm types
Physical (injury)Physical (death)

Business function:
Citizen/customer service

AI system task:
Interaction support/chatbots


Articles about this incident or hazard