Discord AI Moderation Failure Linked to Teen's Death in Brazil

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Discord's machine learning-based moderation system failed to promptly detect and act on a live-streamed incident in which a 13-year-old girl died by suicide in Mato Grosso do Sul, Brazil. Discord's VP of Security admitted the AI system was not fast enough, highlighting ongoing efforts to improve safety models.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions the use of machine learning models (an AI system) to prioritize content for human review in moderation. The failure to detect harmful content in a timely manner on the Discord platform indirectly led to harm (the death of an adolescent). This meets the definition of an AI Incident, as the AI system's use in moderation directly or indirectly contributed to harm to a person. The event is not merely a governance or policy update but relates to an actual harm linked to AI system use.[AI generated]
AI principles
SafetyHuman wellbeing
Industries
Media, social platforms, and marketing
Affected stakeholders
Children
Harm types
Physical (death)
Business function:
Monitoring and quality control
AI system task:
Recognition/object detection

Articles about this incident or hazard