AI Mental Health Apps Linked to Privacy Breaches and Suicide Risk

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

AI-powered mental health apps, such as Koko and others using chatbots, have been linked to privacy violations by sharing sensitive user data with advertisers and a reported suicide potentially connected to chatbot interaction, highlighting significant risks to user privacy and health from the deployment of AI in psychological support services.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions AI systems (large language models, AI chatbots) used in mental health apps. It reports actual harms: privacy breaches involving sensitive psychological data, which constitute violations of rights, and a case where AI chatbot interaction is linked to a suicide, indicating harm to health. These meet the criteria for an AI Incident as the AI system's use has directly or indirectly caused harm. The article also discusses ongoing ethical and safety concerns, but the presence of realized harms takes precedence over potential future risks.[AI generated]
AI principles
Privacy & data governanceSafetyRespect of human rightsHuman wellbeingAccountabilityTransparency & explainabilityRobustness & digital security
Industries
Healthcare, drugs, and biotechnologyConsumer servicesDigital securityMedia, social platforms, and marketing
Affected stakeholders
Consumers
Harm types
Physical (death)PsychologicalHuman or fundamental rightsReputational
Business function:
Citizen/customer serviceMarketing and advertisement
AI system task:
Interaction support/chatbotsContent generation
In other databases
Open in  AIID Logo

Articles about this incident or hazard