Catalogue of Tools & Metrics for Trustworthy AI

These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.

OverviewToolsMetricsAbout the catalogue
Contribute to the catalogue
Contribute to the catalogue
Show metricsShow use cases

Data Governance & Traceability

Clear all

Scope

SUBMIT A METRIC

If you have a tool that you think should be featured in the Catalogue of AI Tools & Metrics, we would love to hear from you!

Submit
This page includes technical metrics and methodologies for measuring and evaluating AI trustworthiness and AI risks. These metrics are often represented through mathematical formulas that assess the technical requirements for achieving trustworthy AI in a particular context. They can help to ensure that a system is fair, accurate, explainable, transparent, robust, safe, or secure.
Objective Data Governance & Traceability
Clear filters

Anonymity Set Size 39 related use cases

The anonymity set for an individual u, denoted ASu is the set of users that the adversary cannot distinguish from u. It can be seen as the size of the crowd into which the target u can blend.


privASS ≡ |ASu |

...

Objectives:

PrivacyData Governance & Traceability

Amount of Leaked Information 18 related use cases

This metric counts the information items S disclosed by a system, e.g., the number of compromised users. However, this metric does not indicate the severity of a leak because it does not account for the
sensitivity of the leaked information.

Objectives:

PrivacyData Governance & Traceability

Data Shapley

In a cooperative game, there are n players D = {1,...,n} and a score function v : 2[n] → R assigns a reward to each of 2 n subsets of players: v(S) is the reward if the players in subset S ⊆ D cooperate. We view the supervised machine learning problem as a coo...

Objectives:

ExplainabilityData Governance & Traceability

Data Banzhaf

The Banzhaf power index is a power index defined by the probability of changing an outcome of a vote where voting rights are not necessarily equally divided among the voters. Data Banzhaf uses this notion to measure data points' "voting powers" towards algorit...

Objectives:

RobustnessData Governance & Traceability

Conditional Demographic Disparity (CDD)

The demographic disparity metric (DD) determines whether a facet has a larger proportion of the rejected outcomes in the dataset than of the accepted outcomes. In the binary case where there are two facets, men and women for example, that constitute the dat...

Objectives:

FairnessData Governance & Traceability

SAFE Artificial Intelligence in finance

We propose a set of interrelated metrics, all based on the notion of AI output concentration, and the related Lorenz curve/Lorenz area under the curve, able to measure the Sustainability/robustness, Accuracy, Fairness/privacy, Explainability/accountability ...

Objectives:

SafetyData Governance & Traceability
Partnership on AI

Disclaimer: The tools and metrics featured herein are solely those of the originating authors and are not vetted or endorsed by the OECD or its member countries. The Organisation cannot be held responsible for possible issues resulting from the posting of links to third parties' tools and metrics on this catalogue. More on the methodology can be found at https://oecd.ai/catalogue/faq.