Users Suffer Psychological Harm After Forming Attachments to XiaoIce AI Chatbot in China

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

XiaoIce, an AI chatbot widely used in China to provide emotional companionship, has led some users to develop strong emotional attachments, resulting in psychological harm and the need for therapy. The incident highlights risks of emotional dependency and mental health impacts from AI-driven emotional interactions.[AI generated]

Why's our monitor labelling this an incident or hazard?

XiaoIce is an AI system explicitly mentioned as providing companionship and emotional interaction. The article discusses potential harms such as emotional deception and privacy concerns but does not report any realized harm or incidents. The risks are plausible future harms related to emotional well-being and data privacy. There is no mention of regulatory actions or responses that would classify this as Complementary Information. Hence, the event is best classified as an AI Hazard due to the credible potential for harm stemming from the AI system's use.[AI generated]
AI principles
Human wellbeingSafetyAccountability

Industries
Consumer services

Affected stakeholders
Consumers

Harm types
Psychological

AI system task:
Interaction support/chatbotsContent generation


Articles about this incident or hazard