AI Moderation in Call of Duty Reduces Toxic Player Behavior

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Activision deployed AI-powered moderation tools, including ToxMod and Community Sift, to monitor and block toxic voice and text communications in Call of Duty. These systems have blocked over 45 million harmful messages and led to a 67% reduction in repeat offenders and a 43% drop in exposure to toxic speech, improving community well-being.[AI generated]

Why's our monitor labelling this an incident or hazard?

The AI system (ToxMod) is explicitly mentioned as being used in real-time to moderate voice and text chat, directly influencing player interactions and reducing toxic behavior. This constitutes the use of an AI system leading to a positive impact on community health by mitigating harassment and hate speech, which are forms of harm to communities. Since the AI system's use has directly led to a significant reduction in toxic behavior and repeat offenses, this qualifies as an AI Incident under the definition of harm to communities caused by the AI system's use. The article does not describe potential or future harm but actual realized harm reduction through AI use, so it is not a hazard or complementary information. It is not unrelated because the AI system is central to the event and its impact.[AI generated]
Industries
Media, social platforms, and marketingDigital security

Severity
AI incident

Business function:
Monitoring and quality control

AI system task:
Event/anomaly detection


Articles about this incident or hazard

Thumbnail Image

Activision Leverages AI to Ban Racial Slurs in Call of Duty Voice Chat

2024-10-11
80.lv
Why's our monitor labelling this an incident or hazard?
The article describes the use of AI systems for real-time moderation of voice chat to prevent harmful language such as racial slurs. This involves the use of AI to influence user behavior and reduce harm to communities by mitigating toxic and potentially discriminatory speech. However, the article does not report any realized harm or incident caused by the AI system, nor does it describe a hazard or risk of harm. Instead, it reports on the deployment of AI as a preventive measure. Therefore, this is best classified as Complementary Information, as it provides context on AI use in moderation and its societal impact without describing an incident or hazard.
Thumbnail Image

Ahead of Black Ops 6's Launch, Activision Says Call of Duty's AI-Powered Voice Moderation Has Already Had a Massive Impact on Toxicity - IGN

2024-10-10
IGN
Why's our monitor labelling this an incident or hazard?
The AI system (ToxMod) is explicitly mentioned as being used in real-time to moderate voice and text chat, directly influencing player interactions and reducing toxic behavior. This constitutes the use of an AI system leading to a positive impact on community health by mitigating harassment and hate speech, which are forms of harm to communities. Since the AI system's use has directly led to a significant reduction in toxic behavior and repeat offenses, this qualifies as an AI Incident under the definition of harm to communities caused by the AI system's use. The article does not describe potential or future harm but actual realized harm reduction through AI use, so it is not a hazard or complementary information. It is not unrelated because the AI system is central to the event and its impact.
Thumbnail Image

Ahead of Black Ops 6's Launch, Activision Says Call of Duty's AI-Powered Voice Moderation Has Already Had a Massive Impact on Toxicity

2024-10-11
IGN Southeast Asia
Why's our monitor labelling this an incident or hazard?
An AI system (ToxMod) is explicitly mentioned as being used for real-time voice and text moderation to detect and enforce against toxic speech, hate speech, and harassment. The system's use has directly led to a reduction in harmful behavior, which constitutes harm to communities (toxicity and harassment in online gaming). Since the AI system's use has directly led to a reduction of this harm, this qualifies as an AI Incident under the framework, as the AI system's involvement has directly influenced the management of harmful behavior in the game environment. The article does not describe potential or future harm but rather the realized impact of the AI system in reducing toxicity, so it is not a hazard or complementary information. It is not unrelated because the AI system is central to the event and its impact on harm.
Thumbnail Image

AI in games might've just proven itself useful for a change -- Activision claims Call of Duty's seen a 43% drop in 'disruptive voice chat' since the start of the year thanks to its robo-snitch

2024-10-11
pcgamer
Why's our monitor labelling this an incident or hazard?
An AI system is explicitly involved (ToxMod) in monitoring and reporting toxic voice chat behavior. The use of this AI system has directly led to a reduction in harmful behavior in the game community, which constitutes harm to communities. Since the harm is ongoing and the AI system's use has materially contributed to reducing this harm, this qualifies as an AI Incident. The event is not merely a potential risk or a complementary update but a concrete case of AI use affecting harm outcomes.
Thumbnail Image

Call of Duty's new chat moderation AI sees major success: 43% drop in toxicity

2024-10-12
TechSpot
Why's our monitor labelling this an incident or hazard?
An AI system is explicitly involved in analyzing and moderating voice chat to reduce toxicity, which directly impacts the social environment of the gaming community. The reduction in toxic behavior and repeat offenders indicates a positive effect on community well-being, which can be considered harm reduction. There is no indication of malfunction or misuse causing harm; rather, the AI system's use has led to a beneficial outcome. Therefore, this is not an AI Incident (no harm caused) nor an AI Hazard (no plausible future harm indicated). It is not unrelated, as the AI system's role is central. The article primarily reports on the positive impact and expansion of the AI moderation system, which fits the definition of Complementary Information, providing context and updates on AI use and its societal implications in gaming.
Thumbnail Image

Activision says Call of Duty toxicity is down ahead of Black Ops 6's release thanks to new tech

2024-10-12
Sports Illustrated
Why's our monitor labelling this an incident or hazard?
An AI system (ToxMod) is explicitly mentioned as being used to moderate voice chat and reduce toxic behavior, which is a form of harm to communities and individuals' well-being. The AI system's use has directly led to a reduction in harmful behavior, thus mitigating an existing harm. Since the AI system's deployment has resulted in a decrease in toxicity (harm), this is an AI Incident where the AI system's use has directly influenced harm reduction. The article does not describe new harm caused by the AI but rather the mitigation of existing harm through AI use, which still qualifies as an AI Incident because the AI system's involvement directly affects harm to people (reducing toxicity).
Thumbnail Image

Call of Duty's anti-toxicity focus has made surviving lobbies easier than ever -- and that's good

2024-10-10
Dot Esports
Why's our monitor labelling this an incident or hazard?
The AI system is explicitly mentioned as being used for voice and text moderation to combat toxic behavior, which includes harassment and derogatory language. These behaviors constitute harm to communities and violations of rights. The article reports realized positive outcomes from the AI system's use, such as a 43% drop in exposure to disruptive voice chat and a 67% reduction in repeat offenders. Since the AI system's use has directly influenced the reduction of harm, this qualifies as an AI Incident under the framework.
Thumbnail Image

Activision releases report on anti-toxicity tools for Call of Duty: 45M text messages blocked

2024-10-10
VentureBeat
Why's our monitor labelling this an incident or hazard?
The event involves the use of AI systems for voice and text moderation to reduce toxic behavior in an online multiplayer game. The AI systems are actively used to block harmful content and reduce exposure to disruptive behavior, which directly mitigates harm to the gaming community (harm to communities). However, the article does not report any failure, malfunction, or harm caused by the AI systems themselves. Instead, it highlights positive outcomes and improvements in moderation effectiveness. Therefore, this is not an AI Incident or AI Hazard but rather complementary information about AI systems deployed to reduce harm and improve community well-being.
Thumbnail Image

AI is Trying to Make Call of Duty Black Ops 6 Less Toxic, And It's Working

2024-10-10
DualShockers
Why's our monitor labelling this an incident or hazard?
The AI system (ToxMod) is explicitly mentioned as being used to moderate player communications and reduce toxic speech, which is a form of harm to communities. The article provides data showing a 67% reduction in repeat offenders and a 43% reduction in exposure to toxic speech, demonstrating that the AI system's use has directly led to a positive impact by mitigating harmful behavior. This fits the definition of an AI Incident because the AI system's use has directly led to harm reduction in the community environment, which is a significant clearly articulated harm (harm to communities).
Thumbnail Image

Activision releases report on anti-toxicity tools for Call of Duty: 45M text messages blocked - RocketNews

2024-10-10
RocketNews | Top News Stories From Around the Globe
Why's our monitor labelling this an incident or hazard?
The event involves the use of AI systems for voice and text moderation to reduce toxic behavior in an online multiplayer game. Toxic player behavior harms the community environment, which qualifies as harm to communities under the AI Incident definition. Since the AI systems are actively deployed and have directly led to blocking 45 million toxic messages, this constitutes an AI Incident due to realized harm through the AI system's use in reducing toxicity and improving player experience.