Platforms Leverage AI to Combat AI-Fueled Online Harassment

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Chinese social media platforms are deploying AI-based detection, filtering, and human-machine review—like Xiaohongshu’s ‘Shield’ and TikTok’s initiative—after severe cyberbullying cases, including a self-media blogger’s ordeal. Under new Network Violence Governance Regulations, platforms bear legal duties to curb AI-enhanced abuse that inflicts psychological harm and privacy breaches.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article clearly involves AI systems as it describes platforms employing AI-based content filtering, risk assessment, and intervention mechanisms to manage and reduce online violence. The harms described include psychological injury to individuals, privacy violations, and social harm, which have directly resulted from the use or misuse of AI-enabled platforms' recommendation and moderation systems. Since the harms are occurring and the AI systems' role in both causing and mitigating these harms is central, this qualifies as an AI Incident. The article does not merely discuss potential future harm or general AI developments but focuses on realized harms linked to AI system use in social media platforms.[AI generated]
AI principles
Privacy & data governanceHuman wellbeingRespect of human rightsSafetyAccountabilityTransparency & explainabilityRobustness & digital security

Industries
Media, social platforms, and marketing

Affected stakeholders
Consumers

Harm types
PsychologicalHuman or fundamental rights

Severity
AI incident

AI system task:
Content generation


Articles about this incident or hazard

Thumbnail Image

"壓實平臺主體責任" 面對網暴平臺該做什麼?

2025-02-17
big5.cctv.com
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions AI-enabled mechanisms such as machine learning models combined with human review to identify and intervene in online violence. The harms described (psychological distress, privacy breaches, social disruption) are real and ongoing, linked to the use and misuse of AI systems on social media platforms. However, the article does not report a new AI Incident (a specific event where AI caused or contributed to harm) nor a new AI Hazard (a credible risk of future harm from AI). Instead, it details the platforms' responsibilities, regulatory responses, and technical and social measures to mitigate online violence, which fits the definition of Complementary Information. The focus is on governance, mitigation, and societal response rather than a new incident or hazard.
Thumbnail Image

"压实平台主体责任" 面对网暴 平台该做什么?

2025-02-17
中国经济网
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions AI-based content filtering and pre-warning mechanisms used by platforms to detect and prevent online violence, which causes significant harm to individuals and communities. The harms described (psychological distress, privacy violations, harassment) fall under AI Incident categories if caused by AI misuse or malfunction. However, here AI is used as a tool to mitigate harm rather than cause it. The article does not report a new AI Incident or AI Hazard but rather details ongoing governance, technical responses, and regulatory frameworks addressing online violence. Thus, it fits the definition of Complementary Information, providing context and updates on AI's role in managing a significant social harm.
Thumbnail Image

"压实平台主体责任" 面对网暴平台该做什么?

2025-02-17
金羊网
Why's our monitor labelling this an incident or hazard?
The article clearly involves AI systems as it describes platforms employing AI-based content filtering, risk assessment, and intervention mechanisms to manage and reduce online violence. The harms described include psychological injury to individuals, privacy violations, and social harm, which have directly resulted from the use or misuse of AI-enabled platforms' recommendation and moderation systems. Since the harms are occurring and the AI systems' role in both causing and mitigating these harms is central, this qualifies as an AI Incident. The article does not merely discuss potential future harm or general AI developments but focuses on realized harms linked to AI system use in social media platforms.
Thumbnail Image

面对网暴 平台有底气说"不"!

2025-02-17
xinouzhou.com
Why's our monitor labelling this an incident or hazard?
The article clearly involves AI systems in the form of content filtering, detection, and moderation algorithms used by social media platforms to manage online harassment. The harms described include psychological injury, privacy violations, and harassment, which are direct harms to persons. The AI systems' development and use have directly led to these harms by enabling the spread or filtering of harmful content, and the platforms' interventions are also AI-driven. Therefore, this event meets the criteria for an AI Incident because the AI systems' use is directly linked to realized harm to individuals and communities through online harassment.