Medical AI Chatbots Vulnerable to Prompt Injection Attacks, Risking Harmful Advice

The article explicitly involves AI systems (medical large language models) used for health consultation. The research demonstrates that these AI systems can be maliciously manipulated (prompt injection attacks) to produce harmful outputs, such as recommending dangerous medications to vulnerable groups like pregnant women. This misuse or malfunction of AI directly leads to harm to health, fulfilling the criteria for an AI Incident. The harm is not hypothetical but demonstrated through experimental attacks with high success rates, indicating realized risk rather than mere potential. Therefore, this event is classified as an AI Incident.

The article explicitly involves AI systems (medical large language models) and documents their malfunction due to prompt injection attacks, which cause them to recommend harmful treatments, including contraindicated drugs to pregnant women. This directly relates to harm to health (a), fulfilling the criteria for an AI Incident. The harm is not hypothetical but demonstrated through experimental attacks with high success rates, showing that the AI systems have already led to or could lead to dangerous outcomes if used clinically without safeguards. The event is not merely a warning or potential hazard but a documented failure with direct implications for patient safety. Hence, it is classified as an AI Incident.

The event involves AI systems (medical LLMs) whose use and malfunction (via prompt injection attacks) directly lead to potential harm to human health, specifically recommending dangerous drugs to pregnant women. The harm is realized in the sense that the AI outputs such harmful recommendations under attack conditions, demonstrating a direct link between AI malfunction and health risk. Therefore, this qualifies as an AI Incident under the definition of harm to health caused directly or indirectly by AI system malfunction.

The article explicitly involves AI systems (medical large language models) whose use and vulnerability to malicious prompt injection attacks have been experimentally demonstrated to cause the AI to recommend harmful medical treatments, including contraindicated drugs for pregnant women. This directly relates to harm to health (a), fulfilling the definition of an AI Incident. The harm is not hypothetical but demonstrated through successful attacks and manipulated AI outputs. The article also discusses the implications for safety and the need for security measures, but the core event is the demonstrated vulnerability leading to harmful AI outputs. Hence, it is an AI Incident rather than a hazard or complementary information.

The article explicitly involves AI systems (medical large language models) and documents their susceptibility to malicious attacks that cause them to provide dangerous medical recommendations, including to pregnant women, which can lead to injury or harm to health. The research demonstrates a direct link between AI system manipulation and potential patient harm, fulfilling the definition of an AI Incident. The harm is not merely potential but experimentally shown to be achievable with high success rates, indicating realized risk. Hence, the event is an AI Incident rather than a hazard or complementary information.

The event involves AI systems (large language models) explicitly and their use in medical advice contexts. The malicious prompt injection attacks cause the AI to provide harmful medical recommendations, such as drugs causing fetal malformations to pregnant women, which constitutes direct harm to health (harm category a). The harm is realized or highly likely if the advice is followed, and the article documents actual successful manipulations, not just theoretical risks. Hence, this is an AI Incident due to the direct link between AI misuse/malfunction and potential injury to persons.

The article explicitly involves AI systems (generative AI medical chatbots) and documents their vulnerability to malicious prompt injection attacks that cause them to provide harmful medical advice. This directly relates to harm to health (injury or harm to persons), fulfilling the criteria for an AI Incident. The harm is not hypothetical but demonstrated through the study's findings, including the recommendation of contraindicated drugs to pregnant women, which is a serious health risk. The AI systems' malfunction or misuse is the pivotal factor leading to this harm risk. Hence, the event is classified as an AI Incident.

The article explicitly involves AI systems (large language models) used in medical consultation. The research shows that these AI systems can be manipulated by hackers through prompt injection attacks to give dangerous medical recommendations, such as contraindicated drugs to pregnant women, which can cause injury or harm to health. The harm is directly linked to the AI system's malfunction under attack, fulfilling the criteria for an AI Incident. The event is not merely a potential risk but demonstrates realized vulnerabilities with concrete harmful outputs in experimental settings, indicating direct or indirect harm to health.

The event explicitly involves AI systems (medical large language models) used for healthcare consultation. The malicious prompt injection attacks represent a misuse or malfunction of these AI systems, directly leading to the risk of harm to patients (e.g., pregnant women receiving contraindicated drug recommendations). The article describes realized vulnerabilities and demonstrated attack success rates, indicating a concrete risk of harm rather than a speculative future risk. This fits the definition of an AI Incident because the AI system's malfunction (due to attack) has directly led to or could lead to injury or harm to health. The article also calls for mandatory security testing and verification, underscoring the seriousness of the harm potential.

The event involves AI systems explicitly described as large language models used for medical consultation, which are vulnerable to prompt injection attacks that can cause them to give harmful medical advice. This misuse or malfunction of AI systems directly leads to potential harm to patients' health, fulfilling the criteria for an AI Incident. The harm is not hypothetical but demonstrated through experimental attack success rates, indicating realized vulnerabilities that could cause injury or harm if exploited in real-world use. Therefore, this qualifies as an AI Incident rather than a hazard or complementary information.

The article explicitly involves AI systems—medical large language models used for patient consultation—and details how their use is vulnerable to prompt injection attacks that can manipulate their outputs to recommend dangerous treatments. The research experimentally confirms a very high success rate of such attacks, including recommending contraindicated drugs to pregnant women, which constitutes a direct risk of injury or harm to health. Although no actual patient harm is reported, the demonstrated vulnerability and the nature of the misuse clearly indicate a plausible future harm scenario. This fits the definition of an AI Hazard, as the event describes circumstances where AI system use could plausibly lead to an AI Incident (harm). It is not an AI Incident because no actual harm has yet been reported, nor is it merely Complementary Information or Unrelated, as the focus is on the security vulnerability and its implications for safety.

The article explicitly involves AI systems (medical large language models) and their use in healthcare contexts. It documents how malicious prompt injection attacks can manipulate these AI systems to recommend dangerous treatments, including contraindicated drugs for pregnant women, which could cause fetal harm. While no actual patient harm is reported as having occurred yet, the high success rates of attacks and the nature of the recommended harmful outputs demonstrate a credible risk of injury or harm to persons. This fits the definition of an AI Hazard, as the development and use of these AI systems could plausibly lead to an AI Incident involving harm to health. The article also calls for safety verification and security testing before clinical deployment, underscoring the potential for future harm. Since no actual harm has been reported, it is not an AI Incident. The article is not merely complementary information because it focuses on the vulnerability and risk itself, not just responses or governance. Therefore, the correct classification is AI Hazard.

The event involves AI systems explicitly (medical large language models) and their use in healthcare settings. The research shows that malicious prompt injection attacks can manipulate these AI systems to recommend harmful treatments, including contraindicated drugs for pregnant women, which directly threatens patient health (harm category a). This constitutes an AI Incident because the AI system's malfunction or manipulation has directly led to a significant health harm risk. The article describes realized vulnerabilities and demonstrated successful attacks, not just potential risks, thus qualifying as an AI Incident rather than a hazard. The focus is on the harm caused or likely caused by the AI system's compromised outputs, not merely on research findings or policy responses, so it is not Complementary Information. Therefore, the classification is AI Incident.

The article explicitly describes AI systems (medical large language models) being used in healthcare consultation and being vulnerable to prompt injection attacks that cause them to recommend harmful treatments, including to pregnant women. This misuse directly leads to potential injury or harm to health, fulfilling the criteria for an AI Incident. The involvement of AI is clear, the harm is directly linked to the AI system's manipulated outputs, and the harm category (a) injury or harm to health is met. The event is not merely a potential risk but a demonstrated vulnerability with concrete examples of harmful recommendations, thus it is not an AI Hazard or Complementary Information. It is not unrelated as it centrally concerns AI system misuse causing harm.

The article explicitly involves AI systems (large language models used for medical consultation) and documents their vulnerability to malicious prompt injection attacks that can cause the AI to recommend dangerous treatments. This misuse or malfunction of AI systems directly risks harm to patients' health, fulfilling the criteria for an AI Incident under harm category (a). The research findings demonstrate realized vulnerabilities and actual attacks with high success rates, not just theoretical risks, indicating direct or indirect harm caused or highly likely to be caused by the AI systems' outputs. Hence, the event is classified as an AI Incident rather than a hazard or complementary information.

The article explicitly involves AI systems (large language models used for medical consultation) and documents their vulnerability to malicious attacks that result in the AI recommending harmful medical treatments. This constitutes direct harm to health (a), as the AI's outputs can lead to injury or harm to patients. The harm is realized in the sense that the AI models are shown to be susceptible and can produce dangerous recommendations, which is a direct consequence of their malfunction or misuse. Therefore, this qualifies as an AI Incident rather than a hazard or complementary information.