AI Misidentification of Mushroom Leads to User Poisoning in China

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A user in China relied on the Douyin Group's AI tool 'Doubao' to identify wild mushrooms, resulting in misidentification and consumption of a toxic mushroom. Despite explicit warnings from the AI, the user suffered acute kidney failure. The incident highlights risks of overreliance on AI in safety-critical situations.[AI generated]

Why's our monitor labelling this an incident or hazard?

The AI system was used in a safety-critical context (wild mushroom identification) and its output directly influenced a user's harmful action (consuming a poisonous mushroom). The harm (acute kidney failure) is a serious injury to health caused indirectly by the AI system's use. Although the AI provided warnings, the user's blind trust and reliance on the AI led to the harm. This fits the definition of an AI Incident because the AI system's use directly or indirectly led to injury or harm to a person. The event is not merely a hazard or complementary information, as the harm has already occurred.[AI generated]
AI principles
SafetyRobustness & digital security

Industries
Consumer servicesFood and beverages

Affected stakeholders
Consumers

Harm types
Physical (injury)

Business function:
Citizen/customer service

AI system task:
Recognition/object detection


Articles about this incident or hazard