FairX

FairX is an open-source benchmarking toolkit for analysing machine learning models in terms of fairness, data utility and explainability. Many methods exist to reduce bias in AI models, but comparing them consistently is difficult. Existing tools also rarely support fair generative models, which create synthetic data designed to reduce bias. FairX addresses this by bringing data loading, bias-mitigation methods and a wide range of evaluations together in one framework. It was presented at the 2nd Workshop on Fairness and Bias in AI at the 2024 European Conference on Artificial Intelligence (ECAI).

Users load one of the built-in datasets or their own, and specify a sensitive attribute, such as sex or race. The built-in datasets include tabular data, such as Adult-Income and COMPAS, and image data, such as CelebA. Users then apply fairness methods at three stages of the machine learning pipeline. Pre-processing removes correlations between features and sensitive attributes. In-processing methods, including the generative models TabFairGAN, DECAF and FairDisco, train models or produce synthetic data under fairness constraints. Post-processing adjusts decision thresholds.

FairX then evaluates the results on several dimensions. Fairness metrics include the demographic parity ratio and equalised odds ratio. Data utility metrics include accuracy, precision, recall, F1-score and AUROC. For synthetic data, FairX also measures how realistic and diverse the data is, and whether it simply copies the original records. Users can then compare trade-offs between fairness and utility across methods. The toolkit is written in Python and released under the MIT licence.

Auto-discovered on 2026-08-05 by OECD Catalogue Automation