JKT48 Issues Legal Warning Over AI-Generated Sexual Content Targeting Members

The article explicitly mentions the malicious use of AI technology to create harmful sexual content involving real individuals, which has caused actual harm to those individuals' reputation and well-being. The AI system's role in generating such content is central to the harm described. The harm includes violations of personal rights and reputational damage, fitting the definition of an AI Incident under violations of human rights or breach of applicable law protecting fundamental rights. The management's response to pursue legal action further confirms the recognition of harm caused by AI misuse.

The article explicitly mentions AI technology being used maliciously to create defamatory and insulting content, including AI-generated pornographic images of JKT48 members, causing real harm to the individuals involved. This constitutes a violation of personal rights and reputational harm, which are harms under the AI Incident definition (violations of human rights or breach of obligations protecting fundamental rights). The AI system's role in generating and disseminating this harmful content is direct and pivotal. Therefore, this event qualifies as an AI Incident rather than a hazard or complementary information.

The article explicitly states that generative AI was used to create sexually explicit images of real people without consent, which is a direct violation of their rights and causes reputational and emotional harm. The involvement of AI in producing harmful content that is being disseminated meets the criteria for an AI Incident under violations of human rights and harm to communities. The event describes actual harm occurring, not just potential harm, and the response involves legal measures to address the issue. Hence, it is classified as an AI Incident.

The article explicitly mentions the creation and dissemination of AI-generated sexual content involving real individuals without consent, causing reputational harm and potential legal violations. The AI system's misuse is directly linked to harm (defamation, insult, and violation of personal rights). Therefore, this event meets the criteria for an AI Incident as it involves realized harm caused by the use of AI technology.

The event involves the use of a generative AI system "Grok" to produce illegal content, including CSAM and unauthorized pornographic images, which are clear violations of law and human rights. The platform's policy and enforcement actions confirm that such harms have occurred or are actively being prevented due to prior incidents. The AI system's misuse is central to the harm, and the article details concrete measures addressing these harms, including legal cooperation and account bans. This fits the definition of an AI Incident because the AI system's use has directly or indirectly led to violations of human rights and harm to communities. The mention of legal responses and organizational actions further supports this classification.

The event explicitly involves the use of generative AI to create manipulated images that harm the members of JKT48, constituting realized harm to their rights and dignity. This meets the criteria for an AI Incident because the AI system's use has directly led to violations of human rights and legal protections (harm category c). The event is not merely a potential risk or a general update but describes actual harm caused by AI misuse.