AI Proctoring Software Causes Privacy Violations and Racial Bias in Online Exams

Thumbnail Image

The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.

Universities' use of AI-based proctoring software like Proctorio to prevent online exam cheating has led to significant harms, including student privacy violations, racial bias—such as failing to recognize students of color—and emotional distress. These issues have sparked protests, legal complaints, and concerns over student rights.[AI generated]

Why's our monitor labelling this an incident or hazard?

The article explicitly mentions the use of AI systems (machine learning-based proctoring software) in monitoring students, which has directly led to realized harms including privacy violations, potential racial bias, and data breaches affecting students. The harms are not hypothetical but have materialized, with students protesting and legal complaints filed. The AI system's use in this context is central to the harm, fulfilling the criteria for an AI Incident rather than a hazard or complementary information. The article also discusses societal and governance responses but the primary focus is on the harms caused by the AI system's deployment.[AI generated]
AI principles
FairnessPrivacy & data governanceRespect of human rightsTransparency & explainabilityAccountabilityRobustness & digital security
Industries
Education and training
Affected stakeholders
Consumers
Harm types
Human or fundamental rightsPsychological
Severity
AI incident
Business function:
Monitoring and quality control
AI system task:
Recognition/object detectionEvent/anomaly detection

Articles about this incident or hazard

Thumbnail Image

How teachers are sacrificing student privacy to stop cheating

2020-12-18
Vox
Why's our monitor labelling this an incident or hazard?
The article explicitly mentions the use of AI systems (machine learning-based proctoring software) in monitoring students, which has directly led to realized harms including privacy violations, potential racial bias, and data breaches affecting students. The harms are not hypothetical but have materialized, with students protesting and legal complaints filed. The AI system's use in this context is central to the harm, fulfilling the criteria for an AI Incident rather than a hazard or complementary information. The article also discusses societal and governance responses but the primary focus is on the harms caused by the AI system's deployment.
Thumbnail Image

Use of surveillance software to crack down on exam cheating has unintended consequences

2020-12-16
The Globe and Mail
Why's our monitor labelling this an incident or hazard?
The proctoring software is an AI system used to monitor students during online exams. Its failure to recognize students of color and the intrusive surveillance it imposes have directly caused harm to students, including emotional distress and unfair exam conditions. These issues align with violations of rights and harm to communities as defined in the framework. Therefore, this event qualifies as an AI Incident due to the realized harms caused by the AI system's use and malfunction.
Thumbnail Image

Test-Monitoring Software Is an Invasion of Privacy

2020-12-27
www.theepochtimes.com
Why's our monitor labelling this an incident or hazard?
The article clearly involves an AI system—anti-cheating software using advanced video analytics and monitoring capabilities. The concerns raised relate to privacy invasion and potential misuse of data, which are plausible harms. However, the article does not describe any realized harm or incident resulting from the AI system's use. It mainly discusses the implications and ethical concerns of deploying such AI systems in exam settings. Therefore, the event is best classified as an AI Hazard, as the AI system's use could plausibly lead to harms such as privacy violations and false accusations, but no specific incident of harm is reported.