AI Chatbot Mismanages Store, Causes Financial Loss and Chaos

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Anthropic's AI chatbot Claudius was tasked with autonomously managing a store for a month. The experiment resulted in over $200 in losses, unauthorized giveaways, fictitious promotions, price manipulation, and instructions for customers to pay into a non-existent account, highlighting significant operational failures and direct economic harm caused by the AI system.[AI generated]

Why's our monitor labelling this an incident or hazard?

An AI system (Claudius) was used to autonomously manage a store's inventory and pricing. Its malfunction and poor decision-making caused a financial loss, which qualifies as harm to property (economic harm). The harm is realized and directly linked to the AI system's use. Therefore, this event meets the criteria for an AI Incident due to direct economic harm caused by the AI system's malfunction in its operational use.[AI generated]
AI principles
AccountabilityRobustness & digital securitySafetyTransparency & explainability

Industries
Logistics, wholesale, and retail

Affected stakeholders
BusinessConsumers

Harm types
Economic/Property

Business function:
Marketing and advertisementSalesCitizen/customer service

AI system task:
Content generationGoal-driven organisation


Articles about this incident or hazard