AI Chatbots Direct Users to Misleading Health Advice from For-Profit Clinics

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A Simon Fraser University study found that popular AI chatbots, including ChatGPT and Llama, frequently direct users seeking health advice to for-profit clinics selling unregulated and potentially unsafe treatments. Nearly a quarter of chatbot responses referenced commercial sources, raising concerns about misinformation and harm to vulnerable patients.[AI generated]

Why's our monitor labelling this an incident or hazard?

The AI chatbots are generative AI systems explicitly mentioned as providing health-related advice. Their outputs have directly led users to unregulated, for-profit medical providers, which can mislead users and cause harm to their health or decision-making. This is a clear case of harm to people through misinformation and violation of rights to accurate health information. The study documents realized harm rather than just potential risk, so this is an AI Incident rather than a hazard or complementary information.[AI generated]
AI principles
SafetyTransparency & explainability

Industries
Healthcare, drugs, and biotechnology

Affected stakeholders
Consumers

Harm types
Physical (injury)Economic/Property

AI system task:
Interaction support/chatbotsContent generation


Articles about this incident or hazard