AI Systems Used for Large-Scale Production and Distribution of Obscene Content Lead to Criminal Convictions in China

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

In China, multiple cases saw AI systems used to generate and distribute obscene content. Developers of the AlienChat app and individuals selling AI tools and tutorials for creating pornographic material were convicted, highlighting the legal and social harm caused by AI misuse and clarifying developer liability boundaries.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event explicitly involves the use of AI software to create pornographic materials, which is illegal and harmful. The defendant sold AI-generated pornography tutorials and software, directly causing harm by enabling the production and dissemination of prohibited content. This constitutes a violation of law and harm to social norms and communities. The AI system's use is central to the incident, fulfilling the criteria for an AI Incident as the AI system's use directly led to harm and legal violations.[AI generated]
AI principles
SafetyAccountability

Industries
Media, social platforms, and marketing

Affected stakeholders
General public

Harm types
Public interest

AI system task:
Content generation


Articles about this incident or hazard