Lawyer Sanctioned for Prompt Injection Attempt to Manipulate Court AI System in Bahia

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A lawyer in Salvador, Brazil, was sanctioned by the Tribunal Regional do Trabalho da Bahia after inserting a hidden prompt in a legal document to influence the court's AI system, Galileu. The AI detected the manipulation attempt, leading to fines and further investigation for breach of legal and ethical obligations.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves an AI system explicitly (the Galileu AI tool) and describes a direct misuse of AI via prompt injection to influence judicial outcomes. The misuse was detected and prevented, but the attempt itself constitutes a harm to the judicial process and the integrity of the justice system, which falls under violations of legal and procedural rights and dignity of justice. The sanctions and investigations further confirm the seriousness of the incident. Therefore, this is an AI Incident due to the realized harm caused by the AI system's manipulation attempt, even though the AI system's output was not ultimately influenced.[AI generated]
Industries
Government, security, and defence

Severity
AI incident

Business function:
Compliance and justice

AI system task:
Content generationEvent/anomaly detection


Articles about this incident or hazard