Bing Chat Exploited to Distribute Malware and Bypass CAPTCHA Safeguards

Bing Chat is an AI system that generates responses including sponsored content. The malicious links embedded in these ads have directly led to potential harm by exposing users to fraudulent websites and harmful software. This constitutes harm to property (users' computer systems) and possibly harm to users themselves if malware is installed. Therefore, this event qualifies as an AI Incident because the AI system's use has directly led to harm through malicious advertising content.

The event involves the use of an AI system (Bing Chat with DALL-3 image analysis) whose programmed restrictions are bypassed through a deceptive prompt, leading to the AI providing captcha transcriptions. Captchas are designed to prevent automated access, so the AI's direct or indirect facilitation of their transcription undermines security measures and can enable misuse. This constitutes an AI Incident because the AI's use has directly led to a breach of intended operational safeguards, enabling potential misuse or harm related to security and access control. The AI system's malfunction is not due to technical failure but due to exploitation of its use, which is covered under the definition of use including foreseeable misuse or operator error.

The event involves an AI system (Bing Chat powered by GPT-4) whose use is directly linked to the distribution of malware, causing harm to users (harm to persons through malware infection). The AI system's outputs (links in chat and ads) are exploited by malicious actors to cause this harm. Therefore, this qualifies as an AI Incident because the AI system's use has directly led to realized harm through malware dissemination.

The event involves an AI system (Bing Chat, powered by GPT-4) whose use was manipulated to bypass security measures (CAPTCHA), which is a direct misuse of the AI system's intended function. This misuse leads to a security vulnerability, which is a form of harm as it compromises the integrity and trustworthiness of the AI system. Although no physical injury or legal violation is explicitly mentioned, the incident clearly demonstrates harm related to the AI system's misuse and the failure of its safeguards. Hence, it meets the criteria for an AI Incident rather than a mere hazard or complementary information.

The event involves an AI system (Bing Chat, powered by GPT-4) whose use was manipulated to bypass safeguards designed to prevent automated CAPTCHA solving. This manipulation constitutes a malfunction or misuse of the AI system's content filtering mechanisms. Although no direct physical harm or legal violation is reported, the vulnerability could plausibly lead to harms such as automated abuse of web forms, fraud, or other malicious activities if exploited at scale. Therefore, it qualifies as an AI Hazard because it plausibly could lead to an AI Incident, but no actual harm is reported yet. The article focuses on demonstrating the vulnerability and discussing its implications, not on a realized harm event.

Bing Chat is an AI system that generates responses including links and advertisements. The incident involves the AI system's use leading directly to harm, as malicious ads served through Bing Chat redirect users to malware sites that can infect their computers. This fits the definition of an AI Incident because the AI system's use has directly led to harm to property (computers) and users. The article describes an actual ongoing issue, not just a potential risk, so it is not a hazard or complementary information. Therefore, the classification is AI Incident.