Xiaohongshu AI Moderation System Wrongfully Flags Compliant Posts

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Xiaohongshu's AI-driven content moderation system, part of its 'Woodpecker' campaign against false promotions, mistakenly flagged many compliant user posts as violations. The error, attributed to system misoperation during a new reporting mechanism rollout, led to wrongful warnings and user complaints, prompting an official apology and urgent system fixes.[AI generated]

Why's our monitor labelling this an incident or hazard?

The '啄木鸟' plan uses AI algorithms combined with human review to detect and act on false promotion content. The incident involves the AI system incorrectly marking legitimate content as violating rules, causing harm to users' rights to free expression and potentially damaging their reputation or community standing. This constitutes an AI Incident because the AI system's malfunction (false positives in content moderation) directly led to harm (wrongful content flagging and user complaints).[AI generated]
AI principles
AccountabilityFairnessTransparency & explainabilityRobustness & digital security

Industries
Media, social platforms, and marketing

Affected stakeholders
Consumers

Harm types
ReputationalPsychologicalHuman or fundamental rights

Severity
AI incident

Business function:
Monitoring and quality control

AI system task:
Event/anomaly detection


Articles about this incident or hazard

Thumbnail Image

不分敌我的"啄木鸟" 小红书对合规笔记错误打标致歉

2020-10-18
凤凰网(凤凰新媒体)
Why's our monitor labelling this an incident or hazard?
The '啄木鸟' plan uses AI algorithms combined with human review to detect and act on false promotion content. The incident involves the AI system incorrectly marking legitimate content as violating rules, causing harm to users' rights to free expression and potentially damaging their reputation or community standing. This constitutes an AI Incident because the AI system's malfunction (false positives in content moderation) directly led to harm (wrongful content flagging and user complaints).
Thumbnail Image

小红书通报"对合规笔记错误打标":误操作,紧急修复中

2020-10-17
和讯网
Why's our monitor labelling this an incident or hazard?
The platform uses an AI system (algorithmic models) for detecting false promotion content. The misapplication of this AI system caused wrongful flagging of compliant posts, which is a direct harm to users' rights and possibly their reputation and platform access. This harm is realized and ongoing, as users report difficulties in appeal and wrongful penalties. Therefore, this event meets the criteria for an AI Incident due to the AI system's malfunction (misoperation) directly leading to harm (violation of user rights and harm to community trust).
Thumbnail Image

不分敌我的"啄木鸟" 小红书对合规笔记错误打标致歉

2020-10-17
phone.cnmo.com
Why's our monitor labelling this an incident or hazard?
An AI system is involved as the content moderation uses 'multi-dimensional algorithm models' combined with human review to detect violations. The system's malfunction (misclassification of compliant content) directly led to harm in the form of wrongful warnings and disruption to users' experience and rights to free expression. This constitutes an AI Incident because the AI system's malfunction caused realized harm to users by incorrectly penalizing their content and undermining trust and platform fairness.