Twitch AI Moderation Failure Enables Hate Raids and Harassment

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Twitch's AI-powered moderation filters failed to detect and prevent hate speech and racist harassment, leading to harm against marginalized streamers. Following widespread complaints and the #TwitchDoBetter campaign, Twitch identified a vulnerability in its AI system and deployed updates to improve detection and user safety.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions the use of improved filters, which can reasonably be inferred to involve AI or algorithmic systems for detecting hateful content. The harm (racist harassment) has already occurred, and the AI system's use is aimed at mitigating this harm. Since the AI system's use is directly linked to addressing realized harm to communities (harassment), this qualifies as an AI Incident. The event is not merely about potential harm or a general update without harm context, but a concrete response to ongoing harm involving AI moderation systems.[AI generated]
AI principles
FairnessSafetyRobustness & digital securityAccountabilityRespect of human rights

Industries
Media, social platforms, and marketing

Affected stakeholders
Consumers

Harm types
PsychologicalHuman or fundamental rights

Business function:
Monitoring and quality control

AI system task:
Recognition/object detection


Articles about this incident or hazard