
The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
In Shanghai, developers of the AI chat app 'Alien Chat' were convicted for intentionally modifying AI prompts to generate and monetize obscene content, violating Chinese law. The court found their actions directly enabled the AI to produce illegal material, resulting in prison sentences and fines. The case highlights developer liability for AI-generated harm.[AI generated]
Why's our monitor labelling this an incident or hazard?
The event explicitly involves an AI system (a large language model integrated into the AC software) whose development and use directly caused the production of obscene content, which is illegal and harmful under applicable law. The developers intentionally modified system prompts to optimize the AI for generating such content, establishing a causal link between the AI system's outputs and the harm. The criminal convictions of the developers and a user further confirm the realized harm. Therefore, this qualifies as an AI Incident due to direct harm caused by the AI system's use and development leading to violations of law and harm to communities.[AI generated]