Lawyers Use Prompt Injection to Manipulate Judicial AI Systems in Brazil

The article explicitly mentions AI systems used in judicial decision support being manipulated through prompt injection, a form of malicious input designed to alter AI outputs. This manipulation has already occurred in multiple cases, leading to fines and investigations, which shows realized harm. The harm includes undermining the fairness and integrity of judicial decisions, which can be considered a violation of legal rights and harm to communities relying on justice. The AI system's use was exploited maliciously, fulfilling the criteria for an AI Incident. The article also discusses ongoing responses and mitigation efforts, but the primary focus is on the incidents themselves, not just complementary information.

The event involves AI systems explicitly used in judicial processes for document analysis and decision support. The malicious prompt injections represent a misuse of AI systems that directly led to attempts to manipulate judicial outcomes, which constitutes a violation of legal rights and undermines the justice system's integrity. The article documents actual occurrences of such manipulations, investigations, and penalties, confirming realized harm. Hence, the event meets the criteria for an AI Incident due to the direct involvement of AI system misuse causing harm to the judicial process and potentially to individuals' rights.

The article explicitly discusses AI systems used in the judiciary to assist in legal decision-making and how hidden commands (prompt injections) have been used by lawyers to manipulate these AI systems. This manipulation has already resulted in concrete harm, such as judicial fines and undermining the justice process, which constitutes a violation of legal rights and judicial fairness. The AI system's involvement is direct and causal in these harms. The article also discusses ongoing risks and responses but the primary focus is on realized incidents of AI misuse causing harm. Hence, this qualifies as an AI Incident under the OECD framework.

The article explicitly involves AI systems used in judicial processes and details how malicious actors inserted hidden commands to manipulate AI outputs, which directly affects judicial decision-making. This manipulation constitutes a violation of legal rights and undermines the justice system, fulfilling the criteria for harm under violations of human rights and breach of legal obligations. The harm is realized, as evidenced by fines, investigations, and judicial responses. Hence, the event is an AI Incident rather than a hazard or complementary information.

The presence of AI systems in judicial analysis is explicit, and the hidden commands were designed to manipulate these AI systems' outputs, directly influencing judicial decisions. The resulting harm includes undermining the fairness and integrity of legal processes, which can be considered a violation of legal rights and a breach of obligations under applicable law. The penalties and investigations confirm that harm has materialized. Hence, this is an AI Incident due to the direct misuse of AI systems causing harm in the judicial context.

The event involves an AI system explicitly used by the judiciary to assist in document analysis. The lawyer's intentional insertion of hidden commands to manipulate the AI's behavior is a misuse of the AI system that directly undermines the fairness and equality of the judicial process, constituting a violation of legal rights and ethical standards. The harm is realized as it compromises the integrity of judicial decision-making. The investigation, penalties, and new guidelines further confirm the seriousness of the incident. Hence, this is an AI Incident rather than a hazard or complementary information.

The event involves the use and attempted misuse of AI systems by lawyers to manipulate judicial outcomes, which directly relates to the use of AI in the judicial system. The prompt injection technique is an AI-specific attack that attempts to alter AI system behavior to produce biased outputs. This manipulation constitutes a breach of legal and ethical standards, violating fundamental rights and the dignity of justice. Since the manipulation attempts were detected and punished, and the AI system's role was pivotal in the incident, this qualifies as an AI Incident under the definitions provided.

The event involves an AI system explicitly mentioned as being used by the judiciary for document processing and administrative support. The lawyer's deliberate insertion of a hidden command to manipulate the AI system's behavior constitutes misuse of the AI system. This misuse directly led to judicial action and sanctions, indicating harm to the integrity of the judicial process, which can be considered a violation of legal obligations and rights. Therefore, this qualifies as an AI Incident because the AI system's use was directly manipulated to cause harm in a legal context.

The article explicitly mentions AI systems used by the court to process legal petitions and how these systems were deliberately manipulated through prompt injection techniques by lawyers inserting hidden commands. This manipulation directly affected judicial outcomes and fairness, constituting a violation of legal rights and undermining the justice system's integrity. The harm is realized, as sanctions and investigations have been initiated. The AI system's misuse is central to the incident, fulfilling the criteria for an AI Incident involving violations of rights and harm to the operation of critical judicial infrastructure.

The article explicitly involves AI systems—large language models or AI tools used to assist judicial decision-making. The prompt injection technique manipulates these AI systems' outputs, leading to indirect harm by biasing judicial support outputs and potentially causing judicial errors. This manipulation compromises the integrity of judicial processes and infrastructure, which is a significant harm to communities and legal rights. Multiple concrete cases with sanctions and disciplinary actions demonstrate realized harm, not just potential risk. Hence, the event meets the criteria for an AI Incident rather than a hazard or complementary information. The governance responses mentioned are secondary to the primary harm described.

The event explicitly involves AI systems used in judicial decision-making processes, which are manipulated through prompt injection attacks. This manipulation leads to distorted judicial outputs, which can cause violations of fundamental rights and undermine the justice system, constituting harm. The article describes realized harm in the form of compromised judicial fairness and legal integrity, not just potential risk. Hence, it meets the criteria for an AI Incident due to indirect harm caused by AI system misuse and malfunction in a critical societal function.

The event explicitly involves AI systems used for automated legal analysis at TJMG. The lawyers' use of hidden commands to influence these AI systems is a misuse of AI, directly impacting judicial fairness and integrity, which are fundamental rights and legal obligations. The resulting penalties and investigations confirm that harm has materialized. Hence, this qualifies as an AI Incident because the AI system's misuse has directly led to harm in the form of compromised judicial processes and potential violations of legal rights.

The article explicitly mentions AI systems used in judicial processes (e.g., the AI system Galileu at the TRT and AI tools at TJMG) being manipulated through prompt injection techniques embedded in legal documents. This manipulation has led to judicial sanctions, ethical investigations, and criminal inquiries, demonstrating direct harm to the legal system's integrity and violation of legal rights. The AI system's misuse is central to the harm, fulfilling the criteria for an AI Incident. The event involves the use and misuse of AI systems leading to violations of rights and harm to the judicial community's trust, which aligns with the definition of an AI Incident.