Clinicians Trust Faulty AI Recommendations Over Experience
2026-07-09
Inside Precision Medicine
The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
A study in Spain found that over 200 physicians often trusted incorrect AI-generated treatment recommendations, even when patient recovery data contradicted the AI's advice. This overreliance on faulty AI outputs, despite evidence to the contrary, highlights significant risks to patient safety in clinical settings.[AI generated]
Why's our monitor labelling this an incident or hazard?
The event involves an AI system providing treatment sensitivity classifications that were inaccurate and ineffective. The clinicians' reliance on these faulty AI recommendations, despite contradictory patient recovery data, indicates that the AI system's use indirectly led to potential harm to patients' health. This fits the definition of an AI Incident because the AI system's use has directly or indirectly led to harm to persons (patients) through erroneous treatment decisions. The study's findings emphasize the real-world risk of harm from AI misuse or overreliance in healthcare, not just a hypothetical hazard or complementary information.[AI generated]