AI-Generated Legal Documents and AI-Enabled Cyberattacks Cause Harm

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

In the US, AI tools like ChatGPT and Claude are generating legal documents with false or misleading information, overwhelming courts with flawed lawsuits. Separately, Google reports the first use of AI by hackers to discover and exploit zero-day software vulnerabilities, escalating cybersecurity threats. Both incidents highlight direct harms caused by AI systems.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions the use of AI chatbots to generate legal briefs that contained fabricated quotes and citations, which were then submitted to courts. This misuse and malfunction of AI systems directly caused harm to individuals (lawyers fined and sanctioned), the legal system (wasted time and resources), and reputational harm to courts and judges. The AI system's role is pivotal as the hallucinations originated from the AI outputs. Hence, this is an AI Incident rather than a hazard or complementary information.[AI generated]
AI principles
AccountabilityRobustness & digital security

Industries
Government, security, and defenceDigital security

Affected stakeholders
GovernmentGeneral public

Harm types
Economic/PropertyPublic interest

AI system task:
Content generationEvent/anomaly detection


Articles about this incident or hazard