AI-Generated Fabricated Citations Undermine Biomedical Research Integrity

The article explicitly states that AI systems used in biomedical research writing are fabricating citations, leading to thousands of papers containing false references. These fabricated citations have not been corrected and can influence clinical guidelines, which doctors use to make treatment decisions, thus indirectly harming patient health. The AI's role in generating these false citations is central to the harm, fulfilling the criteria for an AI Incident involving injury or harm to health (a).

The event involves the use of AI systems (large language models) in the development and use phases, where their outputs (fabricated citations) have directly led to harm in the form of violations of intellectual property rights and harm to communities by undermining scientific integrity and potentially affecting patient health decisions. The harm is realized and ongoing, not merely potential. Therefore, this qualifies as an AI Incident under the framework, as the AI system's role is pivotal in causing significant, clearly articulated harm.

The article explicitly states that AI-generated fabricated citations have appeared in thousands of biomedical papers, influencing clinical guidelines that doctors rely on for patient care. This constitutes harm to health (a) as doctors may make treatment decisions based on false studies. The AI system's malfunction or misuse in generating false citations is directly linked to this harm. The issue is not hypothetical or potential but ongoing and uncorrected, confirming it as an AI Incident rather than a hazard or complementary information.

An AI system was explicitly used to scan and verify references in medical papers, identifying fabricated references. The fabricated references represent a violation of intellectual property and scientific integrity, which can be considered a breach of obligations under applicable law protecting intellectual property rights and potentially harm to health if clinical decisions are based on false information. The AI system's use here is in the development and use phase, directly leading to the identification of harm (fabricated references) that already exists and is increasing. Although the AI system is used to detect the harm rather than cause it, the event centers on the AI system's role in revealing a significant issue affecting medical literature integrity and health outcomes. This fits the definition of an AI Incident because the AI system's use is directly linked to identifying a harm that affects health and intellectual property rights.

The AI system's use in generating fabricated citations has directly led to harm by undermining the reliability of biomedical literature and clinical guidelines, which can result in incorrect medical treatments and harm to patients. The harm is realized and ongoing, as the fabricated citations have not been corrected or retracted and continue to influence healthcare decisions. Therefore, this qualifies as an AI Incident due to the direct link between AI-generated false information and potential harm to health.

The event involves the use and misuse of AI systems (LLMs) in generating scientific papers that contain fabricated citations and errors, which constitute violations of intellectual property and research integrity rights, as well as potential harm to communities through misleading clinical guidelines affecting patient health. The harm is realized and ongoing, as thousands of papers with AI-generated errors have been published and only a small fraction have been corrected or retracted. Therefore, this qualifies as an AI Incident due to the direct and indirect harm caused by AI-generated misinformation in scientific literature.