Facial Recognition Failures Lead to Wrongful Arrests in São Paulo's Smart Sampa System

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

A research report found that São Paulo's Smart Sampa AI surveillance system, using facial recognition, caused false positives, wrongful arrests, and privacy risks. The system, operating since 2023 with up to 40,000 cameras, also exacerbated racial and geographic inequalities, raising concerns about transparency and effectiveness.[AI generated]

Why's our monitor labelling this an incident or hazard?

The Smart Sampa system is an AI system using facial recognition technology. Its use has directly led to wrongful detentions and arrests, which are harms to individuals (injury or harm to persons and violations of rights). The report documents these harms with specific numbers and details, confirming realized harm rather than potential risk. Therefore, this qualifies as an AI Incident because the AI system's malfunction and use have directly caused harm to people and breaches of fundamental rights.[AI generated]
AI principles
FairnessTransparency & explainability

Industries
Government, security, and defence

Affected stakeholders
General public

Harm types
Human or fundamental rightsReputationalPsychological

Business function:
Compliance and justice

AI system task:
Recognition/object detection


Articles about this incident or hazard