Canva AI Tool Replaces 'Palestine' with 'Ukraine' in User Designs, Prompting Apology

The Magic Layers feature is an AI tool designed to decompose images into editable layers, and it malfunctioned by altering specific text content without user consent. This malfunction directly led to harm in the form of distress and potential violation of users' rights to free expression and accurate representation, which can be considered harm to communities or a violation of rights. Since the AI system's malfunction caused realized harm and the company has responded with remediation, this qualifies as an AI Incident rather than a hazard or complementary information.

The AI system (Magic Layers) is explicitly mentioned and is responsible for changing text in user designs without instruction, which is a malfunction. This alteration of the word "Palestine" to "Ukraine" can be seen as a form of censorship or bias, impacting users' rights and causing distress, thus harm to communities. The company confirmed the issue and took remedial action, confirming the AI system's role in causing harm. Therefore, this event meets the criteria for an AI Incident due to the direct harm caused by the AI system's malfunction and biased behavior.

The Magic Layers feature is explicitly described as an AI system that manipulates visual designs by interpreting and separating image elements. The malfunction—swapping "Palestine" with "Ukraine"—is a direct error caused by the AI system's processing. This leads to harm in the form of misinformation and potential distress to affected communities, fulfilling the criteria for harm to communities under the AI Incident definition. Canva's apology and efforts to fix the issue confirm the AI system's role in causing the harm.

The AI system's malfunction directly led to the alteration of user content in a way that misrepresented political terms, which can be considered a violation of rights and harm to communities. The incident involves the use and malfunction of an AI system affecting user-generated content, fulfilling the criteria for an AI Incident. The company's response to fix the issue does not negate the fact that harm occurred due to the AI's malfunction.

The AI system's use led to direct harm by censoring legitimate speech related to political advocacy, which is a violation of human rights (freedom of expression). The harm is realized, not just potential, as users experienced the removal or alteration of the word "Palestine" in their designs. The incident is directly linked to the AI system's content moderation malfunction. Therefore, this qualifies as an AI Incident under the framework, specifically under violations of human rights.