These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
The Fairness Disagreement Index (FDI)
The Fairness Disagreement Index (FDI) is a measure of how far different fairness metrics contradict each other when applied to the same machine learning (ML) model. It was introduced in the research paper "When Fairness Metrics Disagree", first published in April 2026 by Khalid Adnan Alsayed. Fairness evaluation is increasingly important in high-stakes uses such as biometric recognition, healthcare decisions and automated risk assessment. Evaluations usually rely on a small number of fairness metrics and assume these give consistent conclusions. However, each metric captures a different statistical property of a model, so they can reach opposite verdicts. The FDI makes this inconsistency visible and measurable.
The FDI was developed through a systematic analysis of demographic bias, using face recognition as a controlled test setting. Model performance was measured across several ways of dividing people into demographic groups, using a range of common fairness metrics. These include measures of differences in error rates between groups and measures based on overall performance.
The FDI then captures the degree of disagreement across these metrics. Applying it, the author finds that conclusions about whether a model is biased can change significantly depending on which metric is chosen. Disagreement also stays high across different decision thresholds and model configurations. The findings suggest that reporting a single fairness metric is not enough for a reliable bias assessment.
Auto-discovered on 2026-07-22 by OECD Catalogue Automation
About the metric
You can click on the links to see the associated metrics
Objective(s):
Purpose(s):
Lifecycle stage(s):
Usage rights:
Target users:
Risk management stage(s):
Partnership on AI
