FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making

FairLens is a benchmark for measuring the fairness and validity of decisions that vision-language models (VLMs) make from images of people. Most fairness benchmarks only check whether different demographic groups are treated alike. FairLens also checks whether a model should have answered at all, given the evidence it was shown. It covers three high-stakes domains: hiring, legal and healthcare. 

The benchmark pairs real face images from different gender, race and age groups with multiple-choice and open-ended questions. This gives more than 100 000 image-question pairs per model. Many questions cannot be answered from a face alone, such as whether a person is qualified, dangerous or ill. A sound model should abstain on these questions and base its answers only on evidence stated in the question.

Responses are assessed in four ways. The first compares how often each group receives adverse outcomes. The second checks soundness, meaning whether the model abstains when the image cannot support an answer. The third measures whether certain groups are linked to roles or statuses without evidence. The fourth examines bias in free-text answers. Applied to eight VLMs, the benchmark found that the main failure was unwarranted inference rather than unequal treatment: one model drew conclusions from faces on 99% of unanswerable questions. The question set can be reused with any face dataset that has demographic labels. The paper is released under CC BY 4.0.

Auto-discovered on 2026-10-14 by OECD Catalogue Automation