The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
The Bahia State Court (TJBA) in Brazil detected a prompt injection attack, where a lawyer embedded hidden commands in a legal document to manipulate the court's AI tools into granting habeas corpus for a client accused of serious crimes. The attempt was identified and blocked, prompting disciplinary and legal investigations.[AI generated]
Why's our monitor labelling this an incident or hazard?
The event involves an AI system explicitly described as generative AI tools used for document triage and procedural analysis in the court. The prompt injection attack is a deliberate misuse of the AI system to produce biased outputs favoring the defense's request, which is a direct manipulation of the AI's decision-support function. This manipulation attempts to subvert legal processes, constituting a violation of legal and ethical rights and undermining judicial integrity, which fits the definition of harm to rights and communities. Since the manipulation was detected and the habeas corpus denied, harm was averted in terms of judicial outcome, but the attempt itself and the presence of malicious AI manipulation is a realized AI Incident. The disciplinary and criminal responses further confirm the seriousness and realized nature of the harm. Therefore, this event qualifies as an AI Incident.[AI generated]