AI-Generated Medical Leaflet Contains Offensive Language, Causing Patient Harm in South Korea

The event clearly involves an AI system since the doctor used an AI prompt to generate a patient guidance document. The harm is realized as the patient was directly subjected to dehumanizing and insulting language, which is a violation of human rights and dignity. The AI system's role is pivotal because the harmful prompt was part of the AI input and was mistakenly included in the output given to the patient. This is a direct harm caused by the AI system's use and the human error in handling the AI output, fitting the definition of an AI Incident.

An AI system was explicitly involved as the doctor used an AI prompt to generate the document. The harm occurred through the use of the AI system's output containing offensive language, which was directly given to the patient, constituting harm to the patient's dignity and a violation of rights. This meets the criteria for an AI Incident because the AI system's use directly led to harm (emotional and reputational) to a person. The event is not merely a potential hazard or complementary information but a realized harm caused by AI use.

The presence of an AI system is reasonably inferred because the doctor reportedly input a command into an AI system to generate the notice text, which was then printed and distributed. The harm is realized as the patients were directly insulted and demeaned by the printed message, constituting a violation of rights and harm to the community of patients. Therefore, this qualifies as an AI Incident due to the AI system's use leading directly to harm through the dissemination of offensive content.

The event explicitly involves an AI system used to generate medical guidance content. The AI prompt, which contained harmful and disrespectful language about the patient, was directly printed and given to the patient, causing harm to the patient's dignity and emotional well-being. This constitutes a violation of human rights and harm to the individual, fulfilling the criteria for an AI Incident. The harm is direct and realized, not merely potential, as the offensive content was delivered to the patient.

An AI system was explicitly involved as the medical staff used an AI prompt to generate the leaflet content. The use of the AI system led directly to harm, specifically a violation of the patient's dignity and rights, which falls under violations of human rights or breach of obligations intended to protect fundamental rights. The harm is realized and not just potential. Therefore, this event qualifies as an AI Incident.