AI-Generated Sexual Deepfakes Lead to Arrests and Widespread Harm in Japan

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

In Japan, generative AI was used to create and distribute over 500,000 non-consensual sexual deepfake images of celebrities, politicians, and students, resulting in arrests and significant psychological and reputational harm. The incidents highlight growing concerns over AI-facilitated human rights violations and the urgent need for legal and technical safeguards.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly states that an image generation AI was used to create sexually explicit deepfake images resembling female celebrities. The creation and dissemination of these images have directly led to harm, including violations of personal rights and dignity, which fall under violations of human rights and harm to communities. The AI system's use in producing and distributing this harmful content meets the criteria for an AI Incident, as the harm is realized and directly linked to the AI system's outputs.[AI generated]
AI principles
AccountabilityPrivacy & data governanceRespect of human rightsSafety

Industries
Media, social platforms, and marketing

Affected stakeholders
General public

Harm types
PsychologicalReputationalHuman or fundamental rights

Severity
AI incident

AI system task:
Content generation


Articles about this incident or hazard