AI-Generated Sexualized Images Circulate in School Chats, Causing Harm to Minors

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

AI-manipulated sexualized images and videos are being created and shared without consent in school group chats in Germany, leading to psychological harm, bullying, and violation of minors' rights. Experts highlight the inability of youths to assess consequences, prompting calls for stricter EU regulations to protect children online.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions AI-manipulated images being created and shared without consent, causing distress and harm to minors. This constitutes a violation of rights and harm to individuals and communities. The AI system's use in generating fake images that damage reputations and cause bullying is a direct cause of harm. Hence, this qualifies as an AI Incident under the framework, as the AI system's use has directly led to harm.[AI generated]
AI principles
Respect of human rightsSafety

Industries
Education and training

Affected stakeholders
Children

Harm types
PsychologicalHuman or fundamental rights

AI system task:
Content generation


Articles about this incident or hazard