AI-Generated Deepfake Pornography Harms Twitch Streamers and Influencers

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

AI-powered deepfake technology has been used to create and distribute nonconsensual pornographic videos featuring Twitch streamers and influencers, notably targeting women. Victims, including Sweet Anita, report emotional and reputational harm, while the proliferation of such content highlights the ongoing risks and lack of legal protection against AI-driven identity violations.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly involves AI systems used to generate deepfake videos, which have directly led to harm by violating women's rights and causing psychological and reputational damage. This constitutes a violation of human rights and harm to communities, fitting the definition of an AI Incident. The harm is realized and ongoing, not merely potential, and the AI system's role is pivotal in creating these nonconsensual deepfake videos. Therefore, the event qualifies as an AI Incident.[AI generated]
AI principles
AccountabilityFairnessHuman wellbeingPrivacy & data governanceRespect of human rightsRobustness & digital securitySafetyTransparency & explainability

Industries
Media, social platforms, and marketingDigital security

Affected stakeholders
Women

Harm types
PsychologicalReputationalHuman or fundamental rights

Severity
AI incident

AI system task:
Content generation

In other databases


Articles about this incident or hazard

Thumbnail Image

Nonconsensual deepfake porn puts AI in spotlight | CNN Business

2023-02-17
CNN International
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems used to generate deepfake videos, which have directly led to harm by violating women's rights and causing psychological and reputational damage. This constitutes a violation of human rights and harm to communities, fitting the definition of an AI Incident. The harm is realized and ongoing, not merely potential, and the AI system's role is pivotal in creating these nonconsensual deepfake videos. Therefore, the event qualifies as an AI Incident.
Thumbnail Image

They appeared in deepfake porn videos without their consent. Few laws protect them.

2023-02-14
NBC News
Why's our monitor labelling this an incident or hazard?
The event involves AI systems used to generate deepfake videos, which are highly realistic manipulated videos created through AI techniques. The use of these AI systems has directly led to harm, including violations of privacy and consent, emotional and psychological harm to the victims, and broader societal harm through the spread of nonconsensual sexual content. The harm is realized and ongoing, meeting the criteria for an AI Incident. The article does not merely discuss potential risks or responses but documents actual harm caused by AI-generated content.
Thumbnail Image

Sweet Anita Says People "Don't Know What Consent Means" Amidst Deepfake Scandal

2023-02-14
TheGamer
Why's our monitor labelling this an incident or hazard?
The event explicitly involves AI-generated deepfake pornography, which is an AI system generating manipulated content without consent. The harm includes violation of personal rights and consent, which falls under violations of human rights and fundamental rights. The content has been distributed online, causing harm to the individuals involved. The shutdown of the website and legal actions indicate recognition of harm caused. Therefore, this qualifies as an AI Incident due to direct harm caused by the AI system's use.
Thumbnail Image

Twitch Streamers Appear on Deepfake Porn Without Their Consents

2023-02-15
Tech Times
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions AI-generated deepfake pornographic videos using the faces of Twitch streamers without their consent. This involves AI systems generating manipulated content that harms individuals by violating their rights and causing emotional distress. The harm is realized and ongoing, as evidenced by the increasing number of such videos and their widespread viewership. Therefore, this qualifies as an AI Incident due to violations of human rights and harm to persons caused by the AI system's use.
Thumbnail Image

They Appeared in Deepfake Porn Videos Without Their Consent. Few Laws Protect Them.

2023-02-15
NBC Southern California
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions AI-generated deepfake pornography, which involves AI systems manipulating images and videos to create realistic but fake content. The use of these AI systems has directly led to harm by violating the rights and privacy of individuals without their consent, fitting the definition of an AI Incident under violations of human rights or breach of obligations protecting fundamental rights.
Thumbnail Image

Nonconsensual deepfake porn puts AI in spotlight - KION546

2023-02-17
KION546
Why's our monitor labelling this an incident or hazard?
The article explicitly describes AI systems used to create deepfake videos that have been deployed to harm women by placing their faces in pornographic content without consent. This is a direct violation of rights and causes harm to individuals and communities. The harm is realized and ongoing, not merely potential. The involvement of AI in generating these deepfakes is central to the harm described. Therefore, this event qualifies as an AI Incident under the framework, as the AI system's use has directly led to violations of rights and harm to communities.
Thumbnail Image

Deepfakes pose a real threat

2023-02-18
Pipe Dream
Why's our monitor labelling this an incident or hazard?
The article explicitly discusses AI systems (deepfake generation software) used to create manipulated videos that cause direct harm to individuals, including sexual harassment, psychological harm, and violations of rights. The harms are realized and ongoing, not merely potential. The involvement of AI in generating these deepfakes is central to the harm described. Therefore, this event qualifies as an AI Incident due to direct harm caused by AI-generated content violating rights and causing psychological and social harm.
Thumbnail Image

Twitch stars deceptively edited into porn have little legal protection - Internewscast

2023-02-14
Internewscast
Why's our monitor labelling this an incident or hazard?
The event involves the use of AI systems (deepfake generation technology) to create nonconsensual pornographic content, directly causing harm to individuals by violating their rights and causing emotional and reputational damage. The harm is materialized and ongoing, meeting the criteria for an AI Incident. The article details specific cases, the proliferation of such content, and the impact on victims, confirming direct harm linked to AI system use. Therefore, this is classified as an AI Incident.
Thumbnail Image

The Deepfake Dangers Ahead

2023-02-24
freedomsphoenix.com
Why's our monitor labelling this an incident or hazard?
The article focuses on the potential risks and harms that AI-generated deepfakes could cause, such as fraud, harassment, and misinformation, but does not report a concrete event where these harms have materialized. The presence of AI systems (deepfake generation models) is explicit, and the harms described fall under plausible future harms. Therefore, this qualifies as an AI Hazard rather than an AI Incident or Complementary Information.
Thumbnail Image

Essay | The Deepfake Dangers Ahead

2023-02-23
The Wall Street Journal
Why's our monitor labelling this an incident or hazard?
The article clearly involves AI systems, specifically generative AI used to create deepfake images, videos, and audio. It discusses harms that have occurred (e.g., deepfake videos used in propaganda) and plausible future harms (e.g., misinformation campaigns, national security threats). However, the article is primarily an essay outlining the dangers and potential impacts of deepfakes rather than reporting a concrete AI Incident or a narrowly defined AI Hazard event. It also covers societal and governance responses and technical detection efforts, which are complementary information. Since no specific AI Incident or AI Hazard event is described, the article fits best as Complementary Information, providing context and analysis about AI harms and responses.
Thumbnail Image

I was in deepfake porn, fans think it's real -- it can happen to...

2023-02-24
New York Post
Why's our monitor labelling this an incident or hazard?
The event explicitly involves AI systems used to generate deepfake videos, which are non-consensual and harmful to the individuals targeted. The harms include violation of rights (privacy, dignity), psychological harm, reputational damage, and social stigmatization. These harms have already occurred and are ongoing, fulfilling the criteria for an AI Incident. The AI system's use in creating and disseminating deepfake pornography is central to the harm described, making this a direct AI Incident rather than a hazard or complementary information.
Thumbnail Image

Deepfakes worrying threat to democracy in Africa, says report | News24

2023-02-23
news24
Why's our monitor labelling this an incident or hazard?
The article explicitly discusses deepfakes, which are AI-generated synthetic media, being used to spread false information that has already deceived many people in multiple African countries. This misinformation threatens democratic processes and social stability, which qualifies as harm to communities and a violation of rights. The harm is occurring as the deepfakes are actively misleading people and potentially influencing election outcomes. Therefore, this event meets the criteria for an AI Incident due to the realized harm caused by AI-generated disinformation.
Thumbnail Image

What are deepfakes, and why is the world finding it hard to detect them?

2023-02-22
Business Standard
Why's our monitor labelling this an incident or hazard?
The article explicitly describes deepfakes as AI-generated content using advanced AI techniques (Generative Adversarial Networks). It discusses the potential harms of deepfakes, including impersonation and misinformation, which can harm communities and individuals. Although no specific incident of harm is reported, the government's advisory and the need for fact-checking frameworks indicate a credible risk of harm. The AI system's development and use could plausibly lead to AI Incidents, but since no actual harm is reported yet, the event is best classified as an AI Hazard.
Thumbnail Image

I found sick films of me online & fans think it's real - it can happen to anyone

2023-02-24
The Irish Sun
Why's our monitor labelling this an incident or hazard?
The event clearly involves AI systems (deepfake technology) used maliciously to create and distribute non-consensual explicit content, which constitutes a violation of rights and causes harm to individuals and communities. The harm is direct and materialized, including emotional distress, reputational damage, and potential social and economic consequences. Therefore, this qualifies as an AI Incident under the framework, as the AI system's use has directly led to significant harm.
Thumbnail Image

Deepfake porn, election misinfo would be crime under Minnesota bill

2023-02-25
INFORUM
Why's our monitor labelling this an incident or hazard?
The article explicitly references AI systems (deepfake technology and AI-generated content) and the harms they have caused or could cause, such as nonconsensual pornography and election misinformation. However, the article centers on a proposed bill to criminalize these acts rather than describing a particular incident where AI-generated content has directly caused harm in Minnesota. Therefore, the event is best classified as Complementary Information because it provides governance and societal response context to known and potential AI harms rather than reporting a new AI Incident or AI Hazard.
Thumbnail Image

Twitch Makes Stand Against 'Deepfake Pornography' Following Streamer Controversy

2023-03-08
IGN Southeast Asia
Why's our monitor labelling this an incident or hazard?
The event involves the use of deepfake AI technology to create non-consensual pornographic content, which constitutes a violation of rights and harm to individuals (harm to persons). The incident has already occurred, causing harm to the affected streamers. Twitch's policy change is a response to this AI Incident. Therefore, this qualifies as an AI Incident due to realized harm caused by the AI system's misuse.
Thumbnail Image

Twitch will use machine learning to detect trolls: this you should know

2023-03-05
Bullfrag
Why's our monitor labelling this an incident or hazard?
The event involves the use of an AI system (machine learning tool) to detect trolls and prevent harassment on Twitch. However, the article does not report any realized harm or incident caused by the AI system; rather, it describes a new AI-based moderation tool intended to mitigate harm. There is no indication of malfunction, misuse, or harm resulting from the system's deployment. Therefore, this is a development and deployment of an AI system aimed at harm reduction, without any direct or indirect harm reported. This fits the category of Complementary Information, as it provides context on AI use in content moderation and platform governance without describing an AI Incident or AI Hazard.
Thumbnail Image

Twitch takes a harder stance against explicit deepfakes | Engadget

2023-03-07
engadget
Why's our monitor labelling this an incident or hazard?
The event involves the use of AI-generated synthetic content (deepfakes) that has directly led to harm, specifically violations of rights and harassment of individuals (female streamers). The incident of showing non-consensual deepfake images caused harm to the affected individuals, and Twitch's policy update is a response to this harm. Since the AI system (deepfake generation) was used to create harmful content that violated rights and caused community harm, this qualifies as an AI Incident. The article focuses on the harm caused and the policy response, not just the policy update itself, so it is not merely Complementary Information.
Thumbnail Image

Twitch Is Taking Action On Deepfake Porn

2023-03-08
GameSpot
Why's our monitor labelling this an incident or hazard?
The article focuses on Twitch's response to the potential and existing problem of AI-generated deepfake pornography targeting streamers. While the presence of AI-generated synthetic non-consensual imagery is acknowledged as a harm, the article does not describe a specific AI Incident where harm has directly or indirectly occurred on Twitch. Instead, it details policy updates, expert consultations, and educational efforts aimed at preventing and addressing such harms. Therefore, this is best classified as Complementary Information, as it provides governance and societal response to an AI-related harm issue rather than reporting a new incident or hazard.
Thumbnail Image

Twitch says deepfake porn is now grounds for instaban

2023-03-07
The Verge
Why's our monitor labelling this an incident or hazard?
The event involves AI-generated deepfake porn, which is an AI system's misuse causing harm to individuals' rights and dignity. The streamer incident is a concrete example of harm caused by AI-generated content. However, the article's main focus is on Twitch's policy update and enforcement measures, which are governance and societal responses to the prior harm. Therefore, this is Complementary Information as it provides context and updates on responses to an AI Incident rather than reporting a new incident or hazard.
Thumbnail Image

Millions of Twitch users warned they could be banned for breaking tough new rule

2023-03-08
The Sun
Why's our monitor labelling this an incident or hazard?
The event explicitly involves AI systems generating deepfake images, which are synthetic content created by AI. The misuse and sharing of these images have directly led to harm, including violations of individuals' rights to privacy and consent, and harm to the community of streamers on Twitch. Twitch's policy update and banning of users for such behavior indicate recognition of the harm caused. The incident with the streamer viewing and paying for deepfake images confirms that harm has occurred. Hence, this is an AI Incident involving the use and misuse of AI-generated content causing harm.
Thumbnail Image

Twitch starts to address 'deepfake porn', and the first thing to go is that phrase

2023-03-07
pcgamer
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI-generated synthetic content (AI system involvement) that has caused harm to individuals (non-consensual explicit images causing distress to streamers), which qualifies as violations of rights and harm to communities. However, the article primarily discusses Twitch's policy updates and responses to address these harms rather than detailing a new specific AI Incident. The harm described (distress caused by synthetic NCEI) has already occurred, but the article's main focus is on the platform's governance and mitigation efforts. Therefore, this is best classified as Complementary Information, as it provides updates on societal and governance responses to an existing AI-related harm issue rather than reporting a new AI Incident or AI Hazard.
Thumbnail Image

Twitch addresses livestreaming of explicit deepfake content

2023-03-07
Eurogamer.net
Why's our monitor labelling this an incident or hazard?
The event involves the use of AI-generated synthetic non-consensual explicit content (deepfakes) that has already caused harm to individuals by violating their rights and causing distress. The platform's response aims to mitigate ongoing harm and prevent future incidents. Since the AI system's use (deepfake generation) has directly led to violations of rights and harm to individuals, this qualifies as an AI Incident. The article focuses on the harm caused and the platform's response, not merely on general AI developments or future risks, so it is not Complementary Information or an AI Hazard.
Thumbnail Image

Twitch Finally Addresses Porn Deepfake Scandal Over A Month Later

2023-03-07
Kotaku
Why's our monitor labelling this an incident or hazard?
The event describes a specific incident where AI-generated deepfake pornographic content was used non-consensually, causing harm to individuals' rights and emotional well-being, which fits the definition of an AI Incident due to violations of human rights and harm to communities. Twitch's delayed response and policy updates are complementary but the core event is an AI Incident because the harm has already occurred through the use of AI systems to create and distribute synthetic exploitative images.
Thumbnail Image

Twitch Addresses Deepfake Controversy

2023-03-07
Game Rant
Why's our monitor labelling this an incident or hazard?
Deepfake technology is an AI system capable of generating realistic synthetic media. The incident involves the malicious use of this AI system to create explicit content without consent, directly harming the affected individuals. This fits the definition of an AI Incident as the AI system's use has directly led to harm (violation of rights and reputational damage). Twitch's policy updates and educational initiatives are responses to this incident but do not change the classification of the original event.
Thumbnail Image

Twitch Bans Explicit Deepfakes Following Atrioc Controversy

2023-03-07
TheGamer
Why's our monitor labelling this an incident or hazard?
The event involves the use and dissemination of AI-generated deepfake content, which is an AI system capable of generating realistic synthetic media. The harm caused is the violation of individuals' rights through non-consensual exploitation and distribution of explicit deepfake images, which constitutes a breach of fundamental rights and causes significant personal harm. Twitch's policy update is a response to an incident where such harm occurred, indicating that the AI system's misuse has directly led to harm. Therefore, this qualifies as an AI Incident due to the realized harm from the use of AI-generated non-consensual deepfake content.
Thumbnail Image

Twitch now has a tougher stance against broadcasting deepfake porn

2023-03-08
Neowin
Why's our monitor labelling this an incident or hazard?
The article describes Twitch's new stricter policy against deepfake porn, which involves AI-generated synthetic media. While it references a past incident where a streamer unintentionally broadcasted deepfake content, the main focus is on Twitch's policy response and preventive measures. There is no detailed report of a new AI Incident causing harm but rather a governance and safety response to potential harms. Therefore, this is best classified as Complementary Information, as it provides context and updates on societal and platform responses to AI-related harms without describing a new AI Incident or AI Hazard.
Thumbnail Image

Twitch Clamps Down on Pornographic Deepfakes

2023-03-08
Tech Times
Why's our monitor labelling this an incident or hazard?
The event explicitly involves AI systems generating deepfake pornographic images without consent, which constitutes a violation of individuals' rights and causes harm to communities through harassment and objectification. The use of AI deepfake technology has directly led to harm, as evidenced by the controversy involving a Twitch broadcaster and the ongoing vulnerability of affected individuals. Twitch's policy update is a governance response to an existing AI Incident. Therefore, this event qualifies as an AI Incident due to the direct harm caused by AI-generated non-consensual deepfake pornography.
Thumbnail Image

Twitch Takes a Stand Against Deepfake Pornography: Details Here

2023-03-08
Gadgets 360
Why's our monitor labelling this an incident or hazard?
The article centers on Twitch's updated policy and enforcement stance against AI-generated deepfake pornography, which is a known form of harm involving AI systems. However, the article does not describe a new incident of harm occurring or a new hazard emerging; rather, it details the platform's response and governance measures to mitigate such harms. Therefore, this qualifies as Complementary Information, providing context and updates on societal and governance responses to AI-related harms.
Thumbnail Image

Twitch sancionará a los usuarios que compartan videos para adultos con deepfake

2023-03-08
El Comercio Perú
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions the use of AI-generated synthetic images (deepfakes) that were streamed on Twitch, causing significant distress and violating the rights of the individuals whose images were used without consent. This constitutes a violation of personal rights and exploitation, which fits the definition of harm under AI Incident (c) - violations of human rights or breach of obligations protecting fundamental rights. The AI system's use in generating and distributing non-consensual explicit content directly led to harm. The policy update is a response to this incident but does not negate the fact that the incident occurred. Hence, the event is classified as an AI Incident.
Thumbnail Image

Así Twitch busca luchar contra la pornografía en su plataforma

2023-03-08
Semana.com Últimas Noticias de Colombia y el Mundo
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions the use of AI-generated deepfake pornography (NCEI synthetic) that was shared on Twitch, causing harm to the affected streamers by violating their rights and causing emotional distress. The AI system's use in creating and distributing this content directly led to harm, fulfilling the criteria for an AI Incident. The platform's policy update is a response to this incident, but the core event is the harm caused by AI-generated non-consensual explicit imagery, which is a violation of rights and harm to individuals. Hence, the event is classified as an AI Incident.
Thumbnail Image

Twitch actualiza su política: las medidas que toma para luchar contra el porno "deepfake" | Tecnología | La Voz del Interior

2023-03-08
La Voz
Why's our monitor labelling this an incident or hazard?
The event involves an AI system implicitly, as deepfake pornography is generated using AI-based synthetic image generation technologies. The incident described involved the actual use of AI-generated explicit images without consent, causing personal harm and violation of rights to the individuals depicted. Twitch's policy update is a response to this realized harm. Since the AI system's use directly led to harm (violation of personal rights and emotional distress), this qualifies as an AI Incident rather than a hazard or complementary information. The policy update and the incident are closely linked, but the core event is the harm caused by AI-generated non-consensual explicit content.
Thumbnail Image

Twitch actualiza su política para luchar contra la pornografía...

2023-03-08
europa press
Why's our monitor labelling this an incident or hazard?
The event involves an AI system in the form of synthetic image generation (deepfake pornography) that was used maliciously to create and share non-consensual explicit images, directly causing harm to individuals (violation of rights and personal distress). The incident described (January event) is a clear AI Incident as the AI-generated content led to realized harm. The policy update is a response to this incident but the core event is the AI Incident itself involving harm to persons through AI misuse.
Thumbnail Image

Twitch se une a la batalla en contra de los deepfakes pornográficos

2023-03-08
La Opinión Digital
Why's our monitor labelling this an incident or hazard?
The article describes the creation and dissemination of AI-generated deepfake pornographic images without consent, which is a direct violation of individuals' rights and causes harm to the affected streamers. The AI system (deepfake generation) is explicitly involved and has directly led to harm (violation of rights and non-consensual exploitation). Twitch's policy update is a response to this harm but does not negate the fact that the harm has occurred. Hence, this is an AI Incident.
Thumbnail Image

Twitch asegura que el porno deepfake ahora es motivo para instaban y explica el motivo

2023-03-11
FayerWayer
Why's our monitor labelling this an incident or hazard?
The event centers on the use of AI-generated deepfake pornography, which is a direct violation of individuals' rights and causes harm to the affected streamers. The incident involving the streamer displaying such content demonstrates realized harm from AI misuse. Twitch's policy update is a response to this harm, confirming the AI system's role in causing the incident. Hence, it meets the criteria for an AI Incident due to direct harm from AI-generated content and the platform's enforcement actions.