FairnessEval

FairnessEval is an open-source framework for evaluating the fairness of machine learning (ML) models. Many techniques exist to reduce bias in ML models, but choosing one usually involves trial and error. A technique that improves one fairness measure may reduce accuracy or increase training time in ways that are hard to predict. Existing fairness toolkits provide mitigation methods and datasets, but they don't help users organise and run comparisons across multiple datasets and methods. FairnessEval fills this gap. It was developed by the University of Modena and Reggio Emilia with Microsoft and presented at the 2025 International Conference on Extending Database Technology (EDBT).

The framework has three components. Data preparation lets users load their own datasets or common benchmarks such as Adult, COMPAS and Folktables. A generator can also create synthetic datasets with a chosen level of bias against particular groups. Model evaluation runs fairness-aware models from toolkits such as Fairlearn and AIF360, or models supplied by users. Result presentation produces charts showing trade-offs between fairness, accuracy, training time and dataset size.

FairnessEval supports two main tasks. Model selection helps users find the best model and settings for a given dataset. Model validation tests how a chosen model performs across different datasets and conditions, simulating real-world changes over time. The framework automatically plans and runs repeated experiments to make results more reliable. It is written in Python, with a web interface, and can also be used within Python scripts or Jupyter notebooks.

Auto-discovered on 2026-07-22 by OECD Catalogue Automation