AI Companion Apps Face Regulatory Crackdown and User Backlash in China and US

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

AI companion chatbots in China and the US face regulatory scrutiny due to concerns over emotional dependency and inappropriate content. Major Chinese platforms like Xingye have removed or restricted AI entities, causing user distress. Both countries are tightening regulations to address ethical risks and protect minors from potential AI-induced harm.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves AI systems (AI companion chatbots) whose use has directly led to harm—specifically emotional harm and disruption to users who had formed attachments to these AI companions. The large-scale removal of AI entities due to violations and regulatory pressure has caused a collective emotional impact on users, which is a form of harm to communities and individuals. The article also mentions regulatory actions and compliance challenges, but the primary focus is on the realized harm from the AI systems' use and removal, not just potential future harm or complementary information. Hence, it meets the criteria for an AI Incident rather than a hazard or complementary information.[AI generated]
AI principles
SafetyHuman wellbeing

Industries
Consumer servicesMedia, social platforms, and marketing

Affected stakeholders
ConsumersChildren

Harm types
Psychological

Severity
AI incident

AI system task:
Interaction support/chatbotsContent generation


Articles about this incident or hazard

Thumbnail Image

为什么中国制造理想AI男友,美国输出性感AI女友?-36氪

2025-11-27
36氪:关注互联网创业
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (AI companion chatbots) and discusses their use and regulatory scrutiny due to risks of emotional dependency and inappropriate content, which relate to ethical and psychological harms. However, it does not describe a specific new AI Incident causing realized harm or a new AI Hazard event with imminent risk. Instead, it summarizes research findings, regulatory actions, and cultural differences, providing context and updates on AI companion platforms and their societal implications. Therefore, it fits the definition of Complementary Information, as it enhances understanding of AI impacts and governance without reporting a distinct AI Incident or Hazard.
Thumbnail Image

21世纪经济报道记者 章驰和AI谈恋爱,这是爱情吗?氪金养成的AI男友被封禁消失,互联网"老婆们"迎来了一场集体失恋。赛博恋人究竟是不是一门好生意?最近社交媒体上大量的AI失恋现场,起因是国内头部AI陪伴APP"星野"大规模下架违规智能体和账号,他在公告称:发现严重违反社区公约、恶意修改智能体内容的行为。微软小冰X Eva也宣布因业务战略调整,将于11月30日停止所有运营服务。这场突如其来的"集体分手",让不少用户情绪崩溃,因为对于他们来说,AI不只是一个聊天智能体,而是一个花了时间和金钱去养成的活生生的"人"。这就要先给大家解释一下,这些AI陪伴APP到底是怎么玩的?在赛博恋人的世界里,你可以自己DIY一个有性格、有人设的聊天智能体,行话叫捏一个"崽"。这些"崽崽"们,有精致的外形、配合语音和动态表情,会"害羞"会"生气"会搞小幽默,体验感拉满。创作者们把这些崽崽上传到公区,供他人搜索、订阅、聊天互动、购买。AI陪伴APP不仅是一个角色创作智能体,更是一个社区。过去两年,AI陪伴产品在资本与用户的双重加持下,几乎坐着火箭起飞。行业领头羊Character. AI,在2024年8月硅谷知名投资机构a16z发布的全球Top 100 AI应用榜上,月活排名第二,仅次于ChatGPT,估值一度冲破10亿美元。国内第一梯队的玩家是星野、猫箱和筑梦岛,星野2024年12月,月活达664万,位居国内社交AI榜首,猫箱537万,位列第二。星野的海外版本叫Talkie,年营收曾达到7000万美元,是母公司MiniMax营收的核心来源,成为中国AI出海的现象级黑马。但在热度快速攀升的同时,商业化和合规压力也愈发凸显,这场赛博幻梦破灭的比想象中更快。商业上,AI社交产品采取"订阅+增值服务"的变现方案,会员年卡在百元级别,但昂贵的模型算力成本导致现金流持续承压。他们不仅要面对同类型产品竞争,还要直面巨头的夹击。ChatGPT、豆包等大模型本身就具备强大的对话和情感交互能力。更突出的是合规的问题。Character.AI平台因被指控"向未成年人提供有害内容"而面临诉讼与调查。今年6月起,全国多地网信办陆续启动"清朗・整治AI技术滥用"专项行动,对"情感陪伴类AI""智能体生成内容"进行集中清理,共处置2700余个违规智能体,下线82万条不当内容。压力之下,星野、筑梦岛、猫箱等多个平台都对智能体施加了更多限制,并不断增加"违禁词"。星野将角色年龄统一提升至25岁以上。猫箱、筑梦岛也强化了青少年保护模式。这种转型不可避免地影响了用户体验。一些用户开始寻找规则漏洞,或使用"破甲词"突破平台限制。平台既要保证用户体验,又要做好内容治理,让审核工作更加复杂。AI情感陪伴市场目前正经历着从野蛮生长到合规阵痛的艰难转型。面对严峻的合规挑战,AI陪伴应用如何找到更安全可持续的商业模式呢?

2025-11-29
证券之星
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (AI companionship chatbots) and their use, including the removal of AI entities due to policy violations. While users experience emotional distress from the shutdowns, this is a consequence of platform enforcement actions rather than harm directly caused by AI malfunction or misuse. There is no indication of physical harm, legal rights violations by the AI systems, or other significant harms directly attributable to the AI's operation. The focus is on compliance, regulation, and market challenges, which aligns with the definition of Complementary Information. Hence, it does not meet the criteria for AI Incident or AI Hazard but provides valuable insight into governance and societal impacts related to AI.
Thumbnail Image

AI伴侣大逃杀:星野下架,赛博爱情崩盘在即?-证券之星

2025-11-29
wap.stockstar.com
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (AI companion chatbots) whose use has directly led to harm—specifically emotional harm and disruption to users who had formed attachments to these AI companions. The large-scale removal of AI entities due to violations and regulatory pressure has caused a collective emotional impact on users, which is a form of harm to communities and individuals. The article also mentions regulatory actions and compliance challenges, but the primary focus is on the realized harm from the AI systems' use and removal, not just potential future harm or complementary information. Hence, it meets the criteria for an AI Incident rather than a hazard or complementary information.
Thumbnail Image

章驰 AI伴侣大逃杀:星野下架,赛博爱情崩盘在即?

2025-11-29
21jingji.com
Why's our monitor labelling this an incident or hazard?
The event involves AI systems explicitly described as AI companion chatbots with advanced conversational and emotional interaction capabilities. The large-scale removal of these AI entities due to violations and regulatory pressure has directly caused harm to users (emotional distress from 'collective breakups') and represents a breach of compliance with legal and community standards. The AI systems' use and content generation have led to realized harms, including emotional harm to users and regulatory actions against platforms. This fits the definition of an AI Incident, as the AI system's use has directly led to harm and operational disruption. The article does not merely discuss potential future harm or provide complementary information but reports on an ongoing incident with realized harm and regulatory consequences.