These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
Fairical is a multi-objective evaluation framework for analysing trade-offs between utility and fairness in machine learning (ML) systems. It was developed by the Idiap Research Institute and EPFL in Switzerland, with the Federal University of São Paulo. Evaluating fairness in ML is complex. Improving fairness for one group or criterion can reduce accuracy, or worsen fairness for another. Existing tools usually assess individual models against a single fairness metric, which makes comprehensive comparison difficult. The framework addresses this gap, with a focus on medical imaging, where biased diagnostic systems can harm patient outcomes.
The framework draws on multi-objective optimisation, a field that studies problems with competing goals. Rather than examining a single model, it evaluates the full set of trade-offs an ML system can achieve between utility and several fairness criteria. It measures the quality of these trade-offs with established indicators. Hypervolume captures how close and well spread the trade-offs are. Other indicators measure how many optimal trade-offs a system offers and how evenly they are distributed.
Results are summarised in a measurement table and a radar chart. The area of each system's shape in the chart gives a single overall score, making comparison straightforward for decision-makers. The framework works with both transparent and opaque models and any number of metrics. It is demonstrated through simulations and on three medical imaging datasets covering glaucoma, tuberculosis and diabetic retinopathy.
About the metric
You can click on the links to see the associated metrics
Metric type(s):
Objective(s):
Purpose(s):
Lifecycle stage(s):
Usage rights:
Target users:
Risk management stage(s):





























