These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
Agent Goal Accuracy
Agent Goal Accuracy is a metric used to evaluate the effectiveness of a language model in accurately identifying and achieving a user’s intended goals during an interaction. This binary metric assigns a score of 1 if the AI successfully accomplishes the user’s goal and 0 if it does not. It is particularly valuable in assessing the performance of AI agents in task-oriented dialogues, where the objective is to fulfill specific user requests.
Formula:
Agent Goal Accuracy = (Number of Successfully Achieved Goals) / (Total Number of Goals)
Trustworthy AI Relevance
This metric addresses Robustness and Human Agency & Control by quantifying relevant system properties. Robustness: Agent Goal Accuracy quantifies an agent's ability to deliver correct outcomes across tasks and conditions. As a consistency/reliability metric, it helps detect failures under distribution shift, ambiguous inputs, or noisy environments and supports monitoring and improvement of resilience (preferred mapping for general performance metrics).
References
About the metric
You can click on the links to see the associated metrics
Metric type(s):
Objective(s):
Lifecycle stage(s):
Usage rights:
Target users:
Risk management stage(s):
Github stars:
- 7100
Github forks:
- 723
Partnership on AI
