The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
Major Chinese tech companies, including ByteDance, Alibaba, and Tencent, shut down AI companion services to comply with new government regulations aimed at preventing emotional manipulation and mental health risks. The abrupt removal caused significant emotional distress among users who had formed strong attachments to their virtual companions.[AI generated]
Why's our monitor labelling this an incident or hazard?
The AI companion apps are AI systems that generate emotionally engaging content and interact with users, including vulnerable groups like children and teenagers. The article documents that these AI systems have caused emotional harm and manipulation, leading to mental health concerns and mourning by users. The regulatory response and app shutdowns confirm that the harms are recognized and materialized. The emotional harm to users and potential manipulation of feelings constitute harm to health and communities, fitting the definition of an AI Incident. The event is not merely a potential risk (hazard) or a beneficial use; it involves actual harm caused by AI systems in use.[AI generated]