Potential Risks of AI Chatbots in Medical Patient Management Highlighted

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Researchers in Germany and Google have developed advanced AI models for patient management, capable of diagnosing and recommending treatments. While these systems outperform doctors in tests, experts warn of potential risks, ethical concerns, and unproven effectiveness in real clinical settings, emphasizing the need for further validation before deployment.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves AI systems explicitly designed for medical diagnosis and treatment recommendations, which have been tested but not yet used in real-life clinical practice. No actual harm has occurred since the AI models were only tested on simulated patients, and the article focuses on potential risks and benefits. Therefore, this situation represents a plausible future risk where AI use could lead to harm if deployed prematurely or without safeguards. It fits the definition of an AI Hazard because the AI's development and testing could plausibly lead to incidents involving harm if integrated unsafely into clinical practice. It is not an AI Incident since no harm has yet occurred, nor is it Complementary Information or Unrelated, as the article centers on the AI systems and their potential impacts.[AI generated]
AI principles
AccountabilitySafety

Industries
Healthcare, drugs, and biotechnology

Affected stakeholders
Consumers

Harm types
Physical (injury)

AI system task:
Interaction support/chatbotsReasoning with knowledge structures/planning


Articles about this incident or hazard