Dutch Insurer Pressures Doctors to Use Risky AI Triage Apps

The article describes AI-based triage apps that advise patients on medical appointments, which qualifies as AI systems. The financial coercion by the insurer to use these apps despite their questionable medical validity and regulatory compliance creates a credible risk of harm to patients, especially vulnerable populations who may misunderstand or misuse the app. No actual harm or incidents are reported, but the plausible future harm from reliance on these apps is clear. Hence, this is an AI Hazard rather than an AI Incident. The article also discusses governance and policy context but focuses mainly on the risk posed by enforced use of these AI systems.

The digital triage apps qualify as AI systems because they perform automated decision-making to recommend medical triage outcomes. The event describes the use and mandated adoption of these AI systems despite insufficient evidence of safety, leading to concerns about patient harm and compromised clinical judgment. The financial penalties imposed on doctors who refuse to use these apps create indirect pressure that could lead to harm. This fits the definition of an AI Incident, as the AI system's use has directly or indirectly led to potential or actual harm to patient health (harm category a). The article reports ongoing harm and risk, not just potential future harm, so it is not merely an AI Hazard. It is not Complementary Information because the article focuses on the conflict and risks arising from the AI system's use, not on responses or updates to prior incidents. It is not Unrelated because the AI system and its impact are central to the event.

The triage apps are AI systems that influence medical decision-making by assessing patient needs. The financial coercion by the insurer forces doctors to use these apps despite doubts about their safety and regulatory compliance, which can lead to incorrect triage and potential harm to patients' health. This constitutes indirect harm caused by the AI system's use. The event describes realized harm risks and systemic issues affecting healthcare providers and patients, fitting the definition of an AI Incident rather than a mere hazard or complementary information.

The digital triage apps qualify as AI systems because they perform algorithmic decision-making to provide medical advice. The insurer's financial pressure on doctors to use these apps constitutes use of the AI system. The article reports concerns about the apps' safety and effectiveness, implying a credible risk of harm to patients if the apps provide incorrect advice, especially for vulnerable populations. However, the article does not document actual harm or injury resulting from the apps' use, only the potential for such harm. The financial coercion itself is a governance and ethical issue but does not constitute direct harm caused by the AI system. Hence, the event fits the definition of an AI Hazard (plausible future harm) rather than an AI Incident.