AI Platform Civitai Profits from Nonconsensual Deepfake Porn via Bounty System

The event involves an AI system (Civitai) that facilitates the generation of nonconsensual sexual images, which constitutes a violation of human rights and causes harm to individuals (harm to health and dignity). The harm is ongoing and directly linked to the AI system's use and development. The platform profits from this activity and has features that encourage further harmful AI model creation. Therefore, this qualifies as an AI Incident due to realized harm caused by the AI system's use and malfunction in content moderation.

The event involves an AI system (generative AI models for deepfakes) explicitly used to create realistic images of real people, often without their consent, including sexualized content. This use directly leads to violations of human rights, specifically privacy and potentially labor and intellectual property rights, as well as harm to individuals and communities. The platform incentivizes such harmful content creation, making the AI system's role pivotal in causing these harms. Therefore, this qualifies as an AI Incident under the framework.

The platform's active encouragement and incentivization of creating realistic AI deep fakes of real people, especially public figures, involves the use of AI systems for generating synthetic content that can mislead or harm individuals. While the article does not report a specific incident of harm occurring, the nature of deep fakes and their potential for misuse (e.g., misinformation, defamation, privacy violations) makes it plausible that such harms could arise. The event thus fits the definition of an AI Hazard, as the development and promotion of these AI-generated deep fakes could plausibly lead to violations of rights and harm to communities in the future.

The AI system (Civitai) is explicitly described as generating nonconsensual pornographic images of real people, which is a clear violation of human rights and privacy, fulfilling the criteria for harm under (c). The platform profits from this use and facilitates targeted creation of such content, indicating direct involvement of the AI system's use in causing harm. The investment by Andreessen Horowitz is linked to this harmful AI system, confirming the development and use context. Hence, this is an AI Incident rather than a hazard or complementary information.

The platform uses AI models to generate nonconsensual images, which constitutes a violation of human rights and privacy, a breach of applicable laws protecting fundamental rights. The incentivization of such models through bounties directly leads to harm to individuals' dignity and reputation, qualifying as an AI Incident. The involvement of AI in generating these images and the resulting harm to individuals is explicit and direct.

The marketplace uses AI models to generate deepfake images, which are realistic and nonconsensual, targeting real individuals. The platform incentivizes this behavior through bounties, leading to actual harm in terms of privacy violations and potential mental health impacts. The AI system's use is central to the harm, as it enables the creation and dissemination of these images. The harm is occurring, not just potential, and involves violations of rights and harm to individuals and communities. Hence, this is an AI Incident rather than a hazard or complementary information.

The event involves an AI system explicitly (Civitai's AI model marketplace) that is used to create harmful deepfake content without consent, which constitutes a violation of human rights and causes harm to individuals and communities. The monetization and incentivization of such content production further exacerbate the harm. The harm is ongoing and realized, not merely potential, fulfilling the criteria for an AI Incident. The platform's failure to adequately enforce rules or protect victims also contributes to the harm. Hence, the classification as AI Incident is appropriate.