ASR-FAIRBENCH

ASR-FairBench is an online leaderboard that assesses both the accuracy and the fairness of automatic speech recognition (ASR) systems, which convert speech into text. ASR systems often perform less well for some speakers than others, depending on factors such as accent, gender, age or first language. Existing ASR leaderboards rank models only on overall accuracy, measured by word error rate (WER). A model can therefore rank highly while performing poorly for underrepresented groups. ASR-FairBench addresses this gap by evaluating fairness alongside accuracy. It was developed by researchers at IIT Kharagpur and presented at Interspeech 2025.

The leaderboard uses a stratified 10% sample of Meta's Fair-Speech dataset, which contains recordings of voice assistant commands from 593 US participants. Participants reported their own age, gender, ethnicity, socioeconomic background and first language. The sample preserves the demographic balance of the full dataset while greatly reducing evaluation time.

A statistical model estimates how error rates differ across demographic groups. Each demographic category receives a fairness score from 0 to 100, which is reduced when differences between groups are statistically significant. These scores are combined into an overall fairness score. This is then merged with WER into a single Fairness-Adjusted ASR Score (FAAS), and models are rated on a five-level scale from "severely biased" to "exemplarily fair". Users can submit their own ASR models through a web interface and receive a fairness audit within minutes.

Auto-discovered on 2026-09-23 by OECD Catalogue Automation