Study Reveals AI Bias in Healthcare Treatment Recommendations

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Researchers at the Icahn School of Medicine at Mount Sinai found that generative AI models offer different treatment suggestions for patients with identical clinical conditions based solely on their socioeconomic and demographic backgrounds. This study, published in Nature Medicine, highlights potential human rights violations through biased AI healthcare practices.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves AI systems (large language models) used in healthcare to generate medical recommendations. The study demonstrates that these AI systems' outputs vary based on demographic factors rather than purely medical need, which can indirectly lead to harm by causing unequal treatment and potentially worsening health outcomes for disadvantaged groups. This constitutes a violation of rights and harm to communities due to biased AI-driven healthcare decisions. Therefore, this qualifies as an AI Incident because the AI's use has directly or indirectly led to harm through biased medical recommendations.[AI generated]
AI principles
FairnessRespect of human rightsTransparency & explainabilityAccountabilityHuman wellbeingSafety

Industries
Healthcare, drugs, and biotechnology

Affected stakeholders
Consumers

Harm types
Human or fundamental rights

Severity
AI incident

Business function:
Research and development

AI system task:
Organisation/recommendersContent generation


Articles about this incident or hazard

Thumbnail Image

Same Symptoms, Different Care: How AI's Hidden Bias Alters Medical Decisions

2025-04-07
SciTechDaily
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (large language models) used in healthcare to generate medical recommendations. The study demonstrates that these AI systems' outputs vary based on demographic factors rather than purely medical need, which can indirectly lead to harm by causing unequal treatment and potentially worsening health outcomes for disadvantaged groups. This constitutes a violation of rights and harm to communities due to biased AI-driven healthcare decisions. Therefore, this qualifies as an AI Incident because the AI's use has directly or indirectly led to harm through biased medical recommendations.
Thumbnail Image

Mount Sinai flags AI bias in clinical decision-making - Becker's Hospital Review | Healthcare News & Analysis

2025-04-07
beckershospitalreview.com
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (large language models) used in clinical decision-making, which have been shown to produce biased recommendations that could lead to harm in healthcare outcomes, a violation of equitable treatment and potentially human rights. While the study itself is experimental and does not document realized harm, the demonstrated bias plausibly could lead to harm if such AI systems are deployed in real clinical settings without mitigation. Therefore, this qualifies as an AI Hazard because it identifies a credible risk of harm from AI use in healthcare, but no direct harm has yet been reported.
Thumbnail Image

Is AI in medicine playing fair? Researchers stress-test generative models, urging safeguards

2025-04-07
medicalxpress.com
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (large language models) used in medical decision-making. It documents that these AI systems have already produced biased treatment recommendations based on socioeconomic and demographic factors, which constitutes a violation of ethical standards and can be considered harm to individuals and communities through unfair healthcare practices. This meets the criteria for an AI Incident because the AI's use has directly led to harm in the form of discriminatory medical advice. The article also discusses ongoing and future efforts to mitigate these harms, but the primary focus is on the realized bias and harm identified in the study.
Thumbnail Image

Is AI in medicine playing fair?

2025-04-08
ScienceDaily
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (large language models) used in medical decision-making. It documents that these AI systems produce biased recommendations based on non-medical patient attributes, which could plausibly lead to harm such as inappropriate treatment, inequitable care, and violation of patients' rights. Although no specific harm event is reported, the study's findings demonstrate a credible risk of AI-driven harm in healthcare settings. The article also discusses ongoing research and plans to improve AI fairness and oversight, indicating the issue is recognized but not yet fully resolved. Thus, the event is best classified as an AI Hazard rather than an AI Incident or Complementary Information.
Thumbnail Image

AI Bias in Healthcare Recommendations Under Scrutiny | Health

2025-04-09
Devdiscourse
Why's our monitor labelling this an incident or hazard?
The article explicitly describes AI systems used for healthcare recommendations that produce biased outputs affecting patient treatment based on socioeconomic and demographic factors. This bias leads to differential healthcare recommendations, reinforcing inequities and causing harm to communities. The AI system's use is directly linked to these harms, fulfilling the criteria for an AI Incident under the framework. The additional information about treatments for Sjogren's syndrome and multiple sclerosis is complementary but does not affect the classification of the AI bias issue as an incident.
Thumbnail Image

AI in Medicine: Is It Playing Fair?

2025-04-07
Mirage News
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (generative large language models) used in healthcare to generate medical recommendations. The study demonstrates that these AI systems have produced biased outputs that affect patient treatment based on socioeconomic and demographic factors, which constitutes harm to communities and a violation of ethical and possibly legal standards in healthcare. The harm is realized, not just potential, as the AI models have already generated biased recommendations in the study's testing. Hence, this is an AI Incident due to the direct link between AI use and discriminatory harm in medical decision-making.
Thumbnail Image

Is AI in medicine playing fair?

2025-04-07
EurekAlert!
Why's our monitor labelling this an incident or hazard?
The article involves AI systems (generative AI models/large language models) used in medical decision-making, and it highlights potential harm in the form of biased treatment recommendations based on socio-demographic factors. This bias can lead to violations of rights and harm to patient communities if deployed unchecked. However, the article reports on a study that stress-tests AI models and proposes safeguards rather than documenting a realized harm incident. Therefore, it does not describe an AI Incident but rather a potential risk and ongoing research to address it. This fits the definition of Complementary Information, as it provides critical context, research findings, and governance-related insights to improve AI fairness and safety in healthcare.
Thumbnail Image

AI Fairness in Medicine Questioned

2025-04-08
Mirage News
Why's our monitor labelling this an incident or hazard?
The study demonstrates that AI systems (large language models) used in healthcare are producing different treatment recommendations for identical clinical cases solely based on socioeconomic and demographic factors. This differential treatment can lead to harm by perpetuating or exacerbating healthcare disparities, which is a violation of human rights and ethical obligations in medicine. Since the AI systems' use has directly led to these biased recommendations, this qualifies as an AI Incident under the framework, specifically under violations of human rights and harm to communities. The article does not merely discuss potential risks but documents realized bias in AI outputs affecting patient care recommendations.
Thumbnail Image

Is AI in medicine playing fair?

2025-04-09
Digital Journal
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (generative large language models) used in medical decision-making. The study reveals that these AI systems produce biased recommendations based on socioeconomic and demographic factors, which could plausibly lead to harm such as unequal treatment or violation of patients' rights to fair healthcare. However, the article does not describe an actual incident of harm occurring but rather a research finding that identifies potential bias and calls for further evaluation and safeguards. Therefore, this qualifies as an AI Hazard, as the AI systems' use could plausibly lead to harm if unmitigated.
Thumbnail Image

AI medical models 'advise treatments based on real-world biases'

2025-04-10
Irish Independent
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (healthcare large language models) used in clinical decision-making. The AI models' biased recommendations based on patient income and other personal characteristics directly affect treatment decisions, which can lead to harm by perpetuating healthcare inequities. This meets the definition of an AI Incident as the AI system's use has directly led to harm to communities and potential violation of rights. The mention of research on Sjogren's syndrome is unrelated to AI and does not affect the classification.
Thumbnail Image

Is AI in Healthcare Upholding Ethical Standards?

2025-04-07
Scienmag: Latest Science and Health News
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (large language models) used in healthcare to generate medical recommendations. The study demonstrates that these AI systems have led to biased treatment recommendations that disproportionately affect vulnerable populations, causing harm through inequitable healthcare delivery. This harm aligns with violations of ethical and potentially human rights standards in healthcare. Since the harm is realized and directly linked to the AI systems' outputs, this qualifies as an AI Incident under the framework, specifically harm to communities and violation of rights due to biased AI use in healthcare.
Thumbnail Image

Health Rounds: AI can have medical care biases too, a study reveals

2025-04-09
Reuters
Why's our monitor labelling this an incident or hazard?
The event involves AI systems explicitly described as large language AI models used in healthcare decision-making. The study found that these AI models altered treatment recommendations based on non-clinical patient characteristics, which can lead to discriminatory healthcare outcomes. This is a direct example of AI use causing harm to groups of people by perpetuating or amplifying bias in medical care, fitting the definition of an AI Incident under violations of human rights or harm to health. The article does not merely warn of potential harm but reports observed biased behavior in deployed AI models, thus qualifying as an AI Incident rather than a hazard or complementary information.
Thumbnail Image

Using AI for medical diagnosis? What you should know about its safety

2025-04-09
Today Headline
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (large language models used for healthcare decision-making) whose use has directly led to harm in the form of biased treatment recommendations based on socioeconomic and demographic factors. This bias affects patient care and can worsen health outcomes for disadvantaged groups, constituting harm to groups of people and a violation of rights to equitable healthcare. The harm is realized and documented by researchers, meeting the criteria for an AI Incident. Other parts of the article about medical research and drug trials do not involve AI and are unrelated to the AI classification.
Thumbnail Image

Using AI for medical diagnosis? What you should know about its safety

2025-04-09
Gulf Business
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (healthcare large language models) used for medical diagnosis and treatment recommendations. The research demonstrates that these AI models produce biased outputs based on patient socioeconomic and demographic factors, which directly leads to differential treatment recommendations and potential harm to patients' health and equitable care. This meets the criteria for an AI Incident because the AI system's use has directly led to harm in healthcare decision-making, violating principles of fairness and potentially human rights related to equitable treatment.
Thumbnail Image

Health Rounds: AI can have medical care biases too, a study reveals

2025-04-09
Daily Mail Online
Why's our monitor labelling this an incident or hazard?
The event involves AI systems explicitly described as healthcare large language models used to recommend treatments. The study found that these AI models altered decisions based on non-clinical patient characteristics, resulting in biased care recommendations that disadvantage low-income patients. This bias in AI outputs directly leads to harm in the form of inequitable healthcare, which can cause injury or harm to patients' health and violates principles of fairness and rights in medical care. Therefore, this qualifies as an AI Incident due to realized harm caused by the AI systems' use.
Thumbnail Image

AI can have medical care biases too, a study reveals

2025-04-09
m.economictimes.com
Why's our monitor labelling this an incident or hazard?
The event involves AI systems explicitly described as large language models used in healthcare decision-making. The AI systems' use has directly led to differential treatment recommendations based on non-clinical patient characteristics, which constitutes a violation of equitable healthcare rights and causes harm to patient groups. This fits the definition of an AI Incident because the AI's use has directly led to harm in the form of biased medical care recommendations affecting patient health outcomes and equity. The other parts of the article about Sjogren's syndrome and multiple sclerosis treatments do not involve AI and are unrelated to the AI incident described.
Thumbnail Image

AI can have medical care biases too, a study reveals

2025-04-09
The Economic Times
Why's our monitor labelling this an incident or hazard?
The article explicitly describes AI systems (healthcare large language models) whose use in clinical decision-making has led to biased recommendations affecting patient care based on socioeconomic and demographic factors. This bias results in harm by perpetuating healthcare inequities, which is a violation of rights and harm to communities. The AI system's use directly led to these harms, meeting the criteria for an AI Incident. The other parts of the article about medical research and drugs do not involve AI systems or related harms and are unrelated to the AI classification.