Bias in UK's AI System for Detecting Benefits Fraud

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

The UK's AI system for detecting welfare fraud, used by the Department for Work and Pensions, has been found to unfairly target minorities based on age, disability, marital status, and nationality. Internal assessments revealed significant bias, potentially violating human rights by disproportionately selecting certain groups for fraud investigation.[AI generated]

Why's our monitor labelling this an incident or hazard?

An automated, ML-based system used in real-world decision-making has demonstrably produced biased outcomes that harm individuals’ rights by subjecting them to unwarranted scrutiny. This represents a realized discrimination harm tied directly to the system’s use.[AI generated]
AI principles
FairnessRespect of human rightsTransparency & explainabilityAccountability
Industries
Government, security, and defence
Affected stakeholders
General public
Harm types
Human or fundamental rightsPsychologicalReputationalPublic interest
Business function:
Compliance and justiceMonitoring and quality control
AI system task:
Event/anomaly detectionForecasting/predictionOrganisation/recommenders

Articles about this incident or hazard