AI Chatbots Spread False Medical Information After Experiment With Fabricated Disease

The event involves multiple AI chatbots generating and spreading false medical information about a non-existent disease, which is a direct consequence of their training data and response generation processes. This misinformation can harm individuals by misleading them about health conditions, potentially causing inappropriate health actions or anxiety, which constitutes harm to health and communities. The AI systems' outputs are central to the harm, fulfilling the criteria for an AI Incident. Although the original experiment was designed to be low risk, the real-world impact of AI systems treating the fictitious disease as real and disseminating false information is a clear harm. The event also includes responses and mitigation attempts but the primary focus is on the harm caused by AI-generated misinformation.

The event involves AI systems (ChatGPT, Gemini, Microsoft Copilot) that use advanced language models to generate medical advice. The AI systems were misled by fabricated scientific papers and propagated false medical information as if it were true. This misinformation has already caused harm by polluting medical knowledge and potentially misleading patients and doctors, which fits the definition of harm to communities and harm to health. The AI systems' use and malfunction (hallucination and acceptance of false data) directly led to this harm. Hence, this is an AI Incident rather than a hazard or complementary information.

An AI system (large language models/chatbots) is explicitly involved, as it accepted and propagated fabricated medical information. The harm is indirect but real: misinformation about a non-existent disease can mislead users, potentially causing health-related harm or confusion. The event also highlights risks in AI-generated content being taken as authoritative, leading to violations of trust and possibly health harm. Hence, this qualifies as an AI Incident due to the realized harm from AI's role in spreading false medical information.

The event involves the use of AI systems (large language models) in academic writing processes, which directly produce fabricated citations ('hallucinated citations'). These fabricated references constitute a violation of intellectual property rights and academic integrity, harming the scientific community and public trust in science. The harm is ongoing and increasing, with concrete examples and data supporting the presence of these AI-generated false citations in published literature. The article also discusses responses and mitigation efforts, but the primary focus is on the realized harm caused by AI misuse or malfunction. Hence, the classification as an AI Incident is appropriate.

The event involves AI systems (large language models) that have accepted and propagated fabricated medical information, demonstrating a malfunction in their information validation capabilities. While no direct harm has yet occurred, the plausible future harm includes patients receiving incorrect medical advice, leading to health risks and resource misallocation. The article also discusses the broader implications for trust in academic and medical information due to AI hallucinations. Since the harm is potential and not yet realized, and the AI's role is central to the propagation of false information, this qualifies as an AI Hazard rather than an AI Incident. The article also includes responses from AI companies and experts, but the main focus is on the experiment revealing AI vulnerabilities and the plausible risks arising from them.

The event involves AI systems (large language models) that generated and propagated false medical information about a fabricated disease. The AI's outputs led to misinformation spreading among the public and academic communities, with some researchers citing the fake disease in their work. This misinformation can cause harm to individuals and communities by misleading them about health conditions and potentially prompting unnecessary medical actions. The AI's role is pivotal as it accepted and disseminated the false information as factual, thus directly contributing to the harm. Although the false disease was intentionally fabricated as a test, the AI's failure to detect the falsity and its propagation of the misinformation constitute an AI Incident due to realized harm from AI-generated false medical advice and misinformation.

The event involves AI systems (LLMs) that were used and malfunctioned by generating and disseminating false medical information, which was then indirectly linked to harm through misinformation in healthcare contexts and academic literature. The AI's failure to correctly identify the fictitious disease and its propagation of false medical advice constitutes an AI Incident because it directly led to misinformation that can harm public health and trust. The citation of the false information in a medical journal further demonstrates real-world harm resulting from AI-generated misinformation. Therefore, this qualifies as an AI Incident due to realized harm from AI misuse and malfunction in a critical domain (health).