LLMs produce racist output when prompted in African American English
2024-08-28
Nature
The information displayed in the AIM should not be reported as representing the official views of the OECD or of its member countries.
Studies reveal that AI models, including ChatGPT, exhibit bias against African American English (AAE), leading to discriminatory outcomes in job assignments and legal judgments. Researchers from Stanford, University of Chicago, and Allen Institute found that AI models perpetuate racial stereotypes, raising concerns about AI's role in amplifying discrimination.[AI generated]
Why's our monitor labelling this an incident or hazard?
The article explicitly involves AI systems (LLMs) and their use in generating biased, covertly racist outputs. The harm is realized as these biased outputs can negatively affect African American English speakers, especially in sensitive applications like job screening and policing, which implicates violations of human rights and discrimination. This meets the criteria for an AI Incident because the AI's use has directly led to harm through discriminatory outputs. The study's findings confirm the presence of harm rather than just potential harm, so it is not merely a hazard or complementary information.[AI generated]