Chinese Court Sentences Six for Using Jailbroken AI to Produce and Sell Obscene Content

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A former game company programmer, Zheng and his team, modified a generative AI model to bypass safety controls and produce obscene images and videos for profit. Their AI system generated hundreds of thousands of illicit images, leading to their arrest and sentencing by a Zhejiang court for producing and distributing illegal content.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event explicitly involves the use of an AI system (a generative AI model) to produce illegal obscene content, which is a direct harm to communities and a violation of legal and moral standards. The AI system was deliberately manipulated to bypass safety controls, leading to the generation and distribution of harmful content. The legal consequences and court ruling confirm the harm and the AI system's pivotal role in causing it. Hence, this is an AI Incident due to realized harm caused by the AI system's use in criminal activity.[AI generated]
AI principles
SafetyRobustness & digital security
Industries
Media, social platforms, and marketing
Affected stakeholders
General public
Harm types
Public interest
Business function:
Other
AI system task:
Content generation

Articles about this incident or hazard