AI Platform Civitai Profits from Nonconsensual Deepfake Porn via Bounty System

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Civitai, a major AI model-sharing platform backed by Andreessen Horowitz, enables and profits from the creation of nonconsensual deepfake sexual images of real people. Its 'bounties' feature incentivizes users to generate targeted deepfake models, leading to privacy violations and harm to individuals, especially women and private citizens.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves an AI system (Civitai) that facilitates the generation of nonconsensual sexual images, which constitutes a violation of human rights and causes harm to individuals (harm to health and dignity). The harm is ongoing and directly linked to the AI system's use and development. The platform profits from this activity and has features that encourage further harmful AI model creation. Therefore, this qualifies as an AI Incident due to realized harm caused by the AI system's use and malfunction in content moderation.[AI generated]
AI principles
AccountabilityFairnessHuman wellbeingPrivacy & data governanceRespect of human rightsSafety

Industries
Media, social platforms, and marketing

Affected stakeholders
WomenGeneral public

Harm types
PsychologicalReputationalHuman or fundamental rights

Business function:
Other

AI system task:
Content generation


Articles about this incident or hazard