AI Systems in Medical Imaging Can Infer Patient Race, Raising Bias Concerns

The article explicitly involves AI systems (deep learning models) used in medical imaging to predict patient race. The AI's use and behavior directly relate to potential harm by perpetuating racial biases in healthcare decisions, which can lead to inequitable treatment and health disparities. This aligns with harm categories of violations of human rights and harm to communities. Although the article focuses on research findings rather than a specific incident of harm occurring, the demonstrated capability and the concerns raised about clinical deployment indicate realized or ongoing harm risks. Therefore, this qualifies as an AI Incident due to the direct link between AI use and potential or actual harm in healthcare settings.

The AI system's use in medical diagnosis directly led to harm by missing signs of illness in Black patients, which constitutes injury or harm to health. The AI's ability to infer race from X-rays, despite no obvious markers, suggests a bias in its training or operation that affects health outcomes. This meets the criteria for an AI Incident because the AI's use has directly led to harm to a group of people (Black patients).

The article describes an AI system's use and development that can infer sensitive racial information from medical images, which is not explicitly present. This capability can indirectly lead to violations of human rights and harm to patient groups by perpetuating or amplifying racial biases in healthcare. While no direct harm event is reported, the research uncovers a plausible and credible risk of harm if such AI systems are deployed without safeguards. Therefore, this qualifies as an AI Hazard because the AI system's behavior could plausibly lead to significant harm, including discrimination and inequity in medical treatment.

The AI system's ability to predict race from medical images and the associated risk of biased or racist decisions in clinical settings constitutes a direct link to potential harm, specifically violations of human rights and harm to communities. The article discusses realized capabilities of AI that have already been demonstrated and the risks they pose if deployed, indicating a present and ongoing issue rather than a mere potential future risk. Therefore, this qualifies as an AI Incident due to the direct or indirect harm stemming from the AI system's use and its biased outputs in healthcare.

The AI system's use in medical imaging and diagnosis is explicit, and the AI's ability to infer race—information not consciously accessible to human doctors—could lead to biased treatment recommendations. This constitutes a plausible risk of harm to groups of people (racial bias in healthcare), fitting the definition of an AI Hazard. Since the article does not report actual harm or incidents of biased treatment occurring but warns of potential harm, the event is best classified as an AI Hazard rather than an AI Incident. The research and expert caution emphasize the plausible future harm from AI diagnostic tools generating racially biased decisions.