Chinese AI Language Models Polluted by Inappropriate Internet Content, Study Finds

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A study by Tsinghua University, Ant Group, and Nanyang Technological University reveals that large language models like GPT-4o trained on Chinese internet data are heavily polluted with adult, gambling, and spam content. This contamination leads to AI hallucinations, unreliable outputs, and degraded user experience, raising concerns about misinformation and user trust.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article clearly involves AI systems (conversational AI like ChatGPT) influencing human behavior and language use. The reported changes are indirect effects of AI use, with potential long-term societal and cultural harms suggested but not yet realized. Since no direct or indirect harm has occurred yet, but there is a credible plausible risk of future harm (e.g., loss of linguistic diversity, impact on human cognition and social behavior), this fits the definition of an AI Hazard. The article does not describe an AI Incident because no actual harm has been caused or documented. It is not merely complementary information because the main focus is on the potential negative impact of AI on human language and behavior, not on responses or ecosystem updates. Therefore, the correct classification is AI Hazard.[AI generated]
AI principles
AccountabilityRobustness & digital securitySafetyTransparency & explainability

Industries
Media, social platforms, and marketing

Affected stakeholders
ConsumersGeneral public

Harm types
ReputationalPublic interest

Severity
AI hazard

AI system task:
Content generation


Articles about this incident or hazard

Thumbnail Image

专家警告:AI在"渗透"人类的用词与行为

2025-09-04
The Epoch Times
Why's our monitor labelling this an incident or hazard?
The article clearly involves AI systems (conversational AI like ChatGPT) influencing human behavior and language use. The reported changes are indirect effects of AI use, with potential long-term societal and cultural harms suggested but not yet realized. Since no direct or indirect harm has occurred yet, but there is a credible plausible risk of future harm (e.g., loss of linguistic diversity, impact on human cognition and social behavior), this fits the definition of an AI Hazard. The article does not describe an AI Incident because no actual harm has been caused or documented. It is not merely complementary information because the main focus is on the potential negative impact of AI on human language and behavior, not on responses or ecosystem updates. Therefore, the correct classification is AI Hazard.
Thumbnail Image

GPT-4o 见 AV 女优的次数比您好还多 2.6 倍,AI 正在被中文互联网疯狂污染?

2025-09-07
凤凰网(凤凰新媒体)
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (large language models) and their training data. It details how polluted data (containing adult and gambling content) is embedded in the models' token vocabularies, leading to hallucinations and unreliable outputs. This represents a malfunction or problematic use stemming from AI development and training data sourcing. However, the article does not report any specific realized harm such as injury, rights violations, or community harm caused by these AI outputs. Instead, it warns of the plausible future risks and systemic vulnerabilities arising from this data pollution. Hence, it fits the definition of an AI Hazard rather than an AI Incident. It is not Complementary Information because it is not an update or response to a previously known incident, nor is it unrelated as it clearly concerns AI system development and its risks.
Thumbnail Image

GPT-4o 见 AV 女优的次数比「您好」还多 2.6 倍,AI 正在被中文互联网疯狂污染?

2025-09-07
爱范儿
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (large language models like GPT-4o) and discusses their development and use, focusing on the data pollution problem. The presence of polluted tokens and their impact on AI outputs is a malfunction or deficiency in the AI system's training and behavior. While no direct harm (such as injury, rights violations, or property damage) is reported, the article clearly outlines how this pollution could plausibly lead to harms like misinformation, hallucinations, and degraded user trust or experience. The article does not describe a realized incident but warns of a credible risk inherent in the AI system's training data and behavior. Hence, it fits the definition of an AI Hazard rather than an AI Incident or Complementary Information. It is not unrelated because it centers on AI system issues and their implications.
Thumbnail Image

APPSO|GPT-4o 见 AV 女优的次数比「您好」还多 2.6 倍,AI 正在被中文互联网疯狂污染

2025-09-06
China Digital Times
Why's our monitor labelling this an incident or hazard?
The event involves AI systems (large language models) and their development process (training data contamination). Although no direct harm is reported as having occurred, the contamination causes AI hallucinations and unsafe outputs, which plausibly could lead to harms such as misinformation, exposure to harmful content, or violation of user rights. The article focuses on the potential risks and systemic vulnerabilities rather than a concrete realized harm. Hence, it fits the definition of an AI Hazard rather than an AI Incident or Complementary Information. It is not unrelated because it clearly involves AI systems and their impact.
Thumbnail Image

GPT-4o 见 AV 女优的次数比「您好」还多 2.6 倍,AI 正在被中文互联网疯狂污染_手机网易网

2025-09-06
m.163.com
Why's our monitor labelling this an incident or hazard?
The event involves an AI system (large language models like GPT-4o) whose development (training on polluted data) and use (generating hallucinated or inappropriate content) have directly led to harm. The harms include misinformation, degraded user experience, and potential violation of rights to accurate information, which fall under harm to communities and violation of rights. The article provides concrete evidence from research quantifying the extent of data pollution and its effects on AI outputs, demonstrating realized harm rather than just potential risk. Hence, it meets the criteria for an AI Incident rather than a hazard or complementary information.