MetricLib

MetricLib is an open-source Python toolkit for evaluating the quality of datasets used in medical machine learning. It was developed by the Physikalisch-Technische Bundesanstalt (PTB), Germany's national metrology institute. The toolkit puts into practice the METRIC-framework for trustworthy AI in medicine, which defines the dimensions of data quality that matter for medical AI. MetricLib is the software companion to Metric Hub, PTB's online library of data quality metrics. It helps developers check whether training and test data are fit for purpose and identify data quality gaps.

The toolkit organises its metrics into five modules that follow the clusters of the METRIC-framework: measurement process, timeliness, representativeness, informativeness and consistency. It currently implements 19 metrics. These include completeness, sample entropy, duplicates, Wasserstein distance, maximum mean discrepancy, generalised imbalance ratio and demographic parity. It also includes segmentation metrics such as the Dice similarity coefficient and Hausdorff distance. MetricLib processes a range of data modalities in a memory-efficient way.

Users load a dataset, select the metrics relevant to their use case, and generate a data quality report. Aggregated scores make it easier to identify where data quality falls short. Users can add custom metrics to address use-case-specific requirements. Example notebooks show the workflow on the PTB-XL electrocardiogram dataset and on a medical image segmentation use case. The code is available on GitHub under the GPL-2.0 licence.