New York 'MyCity' Chatbot Hallucinating: Incorrect, Misleading Data Shared
2024-03-30
Tech Times
The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
New York City's official MyCity AI chatbot, launched to provide legal and regulatory guidance to businesses, has been found to give dangerously inaccurate and misleading information. The chatbot's errors include advising users to break laws on housing, labor, and business regulations, potentially causing legal violations and harm to individuals and communities.[AI generated]
Why's our monitor labelling this an incident or hazard?
The AI system ('MyCity' chatbot) is explicitly mentioned and is reported to hallucinate, producing incorrect and misleading information about legal and regulatory matters. This misinformation can cause harm to users who rely on it for important decisions, such as eviction rights or discrimination laws. The harm is realized as users receive wrongful information that could lead to legal or personal consequences. Therefore, this qualifies as an AI Incident because the AI system's malfunction directly leads to harm through misinformation and potential violation of rights or legal obligations.[AI generated]