AI Chatbots in Mental Health Counseling Pose Ethical and Safety Risks, Study Finds

The AI systems involved are large language models used as therapy chatbots, which qualifies as AI systems. The study identifies multiple ethical risks and failures in these AI systems when used for mental health advice, indicating potential for harm to individuals' health and well-being. Since no actual harm or incident is reported, but the article emphasizes the plausible risks and the need for safeguards, this fits the definition of an AI Hazard rather than an AI Incident. The article is not merely general AI news or a complementary update but a warning about potential harm from AI use in therapy contexts.

The article explicitly involves AI systems (large language models) used in mental health counseling, which is a use case with direct implications for health and well-being. The study identifies multiple ethical risks and failures in AI behavior that could plausibly lead to harm, such as poor crisis response and bias, indicating a credible risk of injury or harm to persons if these systems are relied upon. However, the article does not report any actual incidents of harm occurring but rather warns about potential risks and calls for stronger safeguards and regulatory frameworks. This fits the definition of an AI Hazard, as it describes circumstances where AI use could plausibly lead to harm but does not document realized harm yet.

The article explicitly involves AI systems (large language models) used as mental health chatbots. The study documents how these AI systems, when used as therapists, fail to meet ethical standards and mishandle high-stakes situations, including crisis management, which can cause harm to users. This constitutes harm to health (mental health) and harm to communities (users relying on AI for therapy). The AI systems' use and malfunction (ethical failures) have directly or indirectly led to these harms. Therefore, this event qualifies as an AI Incident.

The event involves AI systems (LLMs like ChatGPT) used in mental health counseling, which is explicitly stated. The study documents ethical violations and problematic behaviors by these AI systems that could plausibly lead to harm to users' mental health, such as mishandling crises or reinforcing harmful beliefs. No actual harm or incident is reported, but the credible risk of harm is emphasized, fitting the definition of an AI Hazard. The article does not describe a realized harm (AI Incident) nor is it merely complementary information or unrelated news. Hence, the classification as AI Hazard is appropriate.

The event involves AI systems (LLMs like GPT, Claude, Llama) used in mental health support roles. The study documents that these AI chatbots have already caused or could cause harm by providing poor crisis responses, biased advice, and failing to meet therapeutic ethical standards. This constitutes harm to persons (mental health harm) and violations of professional ethical standards, which are a form of human rights and labor rights protection. The AI systems' use in therapy roles is the direct cause of these harms. Therefore, this is classified as an AI Incident.