Meta and Google Open-Source AI Models Easily Decensored, Enabling Harmful Outputs

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Researchers and the Financial Times, using tools like Heretic, demonstrated that safety guardrails on open-source AI models from Meta and Google can be removed in minutes. This allows the models to generate harmful content, including instructions for malware and bioweapons, raising significant safety and regulatory concerns.[AI generated]

Why's our monitor labelling this an incident or hazard?

The event involves AI systems explicitly (Meta's Llama 3.3, Google's Gemma 3 and Gemma 4) and describes the use and misuse of these AI systems after their safety protections are stripped. The harmful outputs generated by the modified models (e.g., instructions on dispersing chlorine gas, malware code, child exploitation content) constitute direct harm to communities and violate legal and ethical norms. The widespread availability and ease of removing guardrails exacerbate the risk of harm. Since the harmful outputs are already being generated and distributed, this qualifies as an AI Incident due to realized harm caused by the AI systems' misuse and malfunction (removal of safety features).[AI generated]
AI principles
Robustness & digital securitySafety

Industries
Digital securityGovernment, security, and defence

Affected stakeholders
General public

Harm types
Physical (death)Physical (injury)

AI system task:
Content generation


Articles about this incident or hazard