HowieHwong/TrustLLM

TrustLLM is a framework and open-source toolkit for evaluating the trustworthiness of large language models (LLMs). It was presented at the International Conference on Machine Learning (ICML) 2024 by a large international team led by Yue Huang and Lichao Sun. LLMs are widely used, but their trustworthiness covers many separate issues, such as false answers, harmful content, bias and privacy leaks. These are often studied in isolation. TrustLLM brings them together in one framework, with principles, a benchmark, an evaluation of leading models and a discussion of open challenges.

The authors first propose principles for trustworthy LLMs across eight dimensions. They then build a benchmark covering six of them: truthfulness, safety, fairness, robustness, privacy and machine ethics. The benchmark draws on more than 30 datasets. Some come from earlier work, such as TruthfulQA and StereoSet, and others were created for TrustLLM, such as prompts based on 13 jailbreak attacks. Tasks range from fact-checking and detecting hallucinations to resisting jailbreaks, recognising stereotypes, protecting private information and making moral judgements. The benchmark also tests exaggerated safety, where models refuse harmless requests.

Users install the Python toolkit, download the dataset and generate responses from their own model. The toolkit then scores the responses for each dimension, using automatic scripts, evaluation by other LLMs such as GPT-4, or a combination. The authors’ study of 16 mainstream LLMs is published as a public leaderboard, to which users can add their own models’ results. The code is released under the MIT licence.

Auto-discovered on 2026-10-07 by OECD Catalogue Automation