Federal Judge Bans Unverified AI-Generated Legal Filings After ChatGPT Hallucination Incident

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Attorney Steven Schwartz used ChatGPT to draft legal filings, resulting in fabricated case citations submitted to a federal court. Judge Brantley Starr responded by banning unverified AI-generated content in his courtroom, requiring lawyers to certify that filings are either AI-free or thoroughly fact-checked by humans to prevent future misinformation.[AI generated]

Why's our monitor labelling this an incident or hazard?

The AI system (ChatGPT) was explicitly used in the development of legal documents that contained false information, directly leading to harm in the form of misleading the court and potentially violating the rights of involved parties. This is a clear AI Incident because the AI's outputs were relied upon and caused harm in a critical societal domain (justice system). The article also includes complementary information about calls for regulation, but the primary focus is the incident itself involving misuse of AI-generated content in court.[AI generated]
AI principles
AccountabilityTransparency & explainabilityRobustness & digital securitySafetyRespect of human rightsDemocracy & human autonomy

Industries
Government, security, and defence

Affected stakeholders
GovernmentGeneral public

Harm types
ReputationalPublic interest

Business function:
Compliance and justice

AI system task:
Content generationInteraction support/chatbots

In other databases

Articles about this incident or hazard