Research Warns Apple's AI Child Safety Features May Backfire

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Apple's planned AI-powered child safety features, which scan images and issue warnings to deter sharing of explicit content, may inadvertently increase risky behavior among children, according to new research. Experts caution that these design elements could backfire, potentially leading to greater harm rather than prevention.[AI generated]

Why's our monitor labelling this an incident or hazard?

Apple's child safety tools involve AI systems for detecting and limiting CSAM and explicit content. The article focuses on research suggesting these features might inadvertently increase risky behavior, indicating a plausible future harm. However, no actual harm or incident has been reported yet. Therefore, this situation fits the definition of an AI Hazard, as the AI system's use could plausibly lead to harm, but no direct or indirect harm has materialized at this point.[AI generated]
AI principles
Safety

Industries
Consumer products

Affected stakeholders
Children

Harm types
Psychological

AI system task:
Recognition/object detection


Articles about this incident or hazard