Catalogue of Tools & Metrics for Trustworthy AI

These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.

OverviewToolsMetricsAbout the catalogue
Contribute to the catalogue
Contribute to the catalogue
Show metricsShow use cases

Clear all

Safety

Scope

SUBMIT A METRIC

If you have a metric that you think should be featured in the Catalogue of Tools & Metrics for Trustworthy AI, we would love to hear from you !

Submit
This page includes technical metrics and methodologies for measuring and evaluating AI trustworthiness and AI risks. These metrics are often represented through mathematical formulas that assess the technical requirements for achieving trustworthy AI in a particular context. They can help to ensure that a system is fair, accurate, explainable, transparent, robust, safe, or secure.
Objective Safety
Clear filters

Accuracy 175 related use cases

Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:

Accuracy = (TP + TN) / (TP + TN + FP + FN) , where:

TP: True positive

TN: True negative

FP: False positive

FN...

Objectives:

RobustnessSafetyTransparency

SAFE Artificial Intelligence in finance

We propose a set of interrelated metrics, all based on the notion of AI output concentration, and the related Lorenz curve/Lorenz area under the curve, able to measure the Sustainability/robustness, Accuracy, Fairness/privacy, Explainability/accountability ...

Objectives:

SafetyData Governance & Traceability

ARML (Adversarially Robust Metric Learning)

ARML (Adversarially Robust Metric Learning) is a method for learning distance measures that are provably robust to adversarial attacks. It was presented in the paper "Provably Robust Metric Learning" at the 2020 Conference on Neural Information Processing S...

Objectives:

RobustnessSafetyDigital Security

WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

WATCH (Weighted Adaptive Testing for Changepoint Hypotheses) is a statistical monitoring framework for detecting unsafe behavior in deployed AI/ML systems. The method introduces weighted-conformal test martingales (WCTMs), a generalisation of conformal test...

Objectives:

RobustnessSafety

Unveiling Covert Toxicity in Multimodal Data via Toxicity Association Graphs: A Graph-Based Metric and Interpretable Detection Framework

This tool provides a graph-based metric and interpretable detection framework to identify covert toxicity that can arise when multimodal signals are combined. It measures toxicity associations across modalities and produces results that can be interpreted t...

Objectives:

SafetyExplainability
Partnership on AI

Disclaimer: The tools and metrics featured herein are solely those of the originating authors and are not vetted or endorsed by the OECD or its member countries. The Organisation cannot be held responsible for possible issues resulting from the posting of links to third parties' tools and metrics on this catalogue. More on the methodology can be found at https://oecd.ai/catalogue/faq.