AI-Generated Deepfake of Missing Child Causes Distress and Misinformation in Florence

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

AI-generated deepfake videos featuring missing child Kata circulated on TikTok, using synthetic voice and animation to mimic her appeal for help. The videos misled viewers, caused emotional distress, and spread misinformation, prompting calls from experts for regulation of such technology due to its potential for harm.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event explicitly describes an AI system generating a fake video and voice of a missing child, which is being disseminated and believed by some users, causing emotional harm and misinformation. This fits the definition of an AI Incident because the AI system's use has directly led to harm to communities and individuals (emotional distress, misinformation). The harm is realized, not just potential, and the AI system's role is pivotal in creating the false content. Therefore, this is classified as an AI Incident.[AI generated]
AI principles
Transparency & explainabilityPrivacy & data governanceRespect of human rightsSafetyAccountabilityRobustness & digital securityHuman wellbeing

Industries
Media, social platforms, and marketingDigital securityGovernment, security, and defence

Affected stakeholders
ChildrenGeneral public

Harm types
PsychologicalHuman or fundamental rightsReputational

Severity
AI incident

AI system task:
Content generation


Articles about this incident or hazard

Thumbnail Image

Kata 'parla' in un falso video su Tik Tok generato con AI - Cronaca

2023-06-21
ANSA.it
Why's our monitor labelling this an incident or hazard?
The event explicitly describes an AI system generating a fake video and voice of a missing child, which is being disseminated and believed by some users, causing emotional harm and misinformation. This fits the definition of an AI Incident because the AI system's use has directly led to harm to communities and individuals (emotional distress, misinformation). The harm is realized, not just potential, and the AI system's role is pivotal in creating the false content. Therefore, this is classified as an AI Incident.
Thumbnail Image

Kata scomparsa e il video su TikTok con lei che parla: ma è un fake generato dall'intelligenza artificiale

2023-06-23
Virgilio Notizie
Why's our monitor labelling this an incident or hazard?
The video is explicitly described as AI-generated fake content that misrepresents a missing child, causing emotional distress and misleading viewers. This is a direct use of AI generative technology leading to harm (emotional and social), fulfilling the criteria for an AI Incident. The harm is realized, not just potential, and the AI system's role is pivotal in creating the fake video. Therefore, this event is classified as an AI Incident.
Thumbnail Image

"Voglio tornare a casa, mi manca mia mamma", l'appello di Kata è un deepfake: l'ultima macabra tendenza nella cronaca nera - L'Unità

2023-06-22
L'Unità
Why's our monitor labelling this an incident or hazard?
The event involves an AI system (deepfake generation) used to create a synthetic video of a missing child, which misleads viewers and causes emotional distress. Although the video is spreading and causing confusion, the article does not document any direct injury, legal violation, or disruption caused by the AI-generated content. The harm is potential and plausible, as such deepfakes can interfere with investigations, cause emotional harm, or spread misinformation. Hence, this qualifies as an AI Hazard rather than an AI Incident. It is not Complementary Information because the main focus is on the AI-generated deepfake and its implications, not on responses or updates to a prior incident. It is not Unrelated because AI is central to the event.
Thumbnail Image

Bambina scomparsa a Firenze, il video della vergogna con l'intelligenza artificiale. L'esperto: "Macabro e inutile, andrebbero regolamentati" - Video Il Tirreno

2023-06-20
Il Tirreno
Why's our monitor labelling this an incident or hazard?
The event describes the use of AI generative models to create synthetic videos of missing persons, which are then disseminated publicly. This involves AI system use and leads to emotional harm and misinformation, which can be considered harm to communities. While no direct physical injury or legal violation is reported, the AI-generated content causes significant emotional distress and social harm. Therefore, this qualifies as an AI Incident due to the realized harm caused by the AI system's use in generating misleading and emotionally manipulative content. The expert's call for regulation further supports the recognition of harm.