SafeDialBench

SafeDialBench is a benchmark for evaluating the safety of large language models (LLMs) in multi-turn conversations under jailbreak attacks. Jailbreak attacks try to trick a model into producing harmful content. Most existing safety benchmarks test only single-turn exchanges or a single attack method. Yet in real use, harmful requests are often built up gradually over several turns. Existing benchmarks also do not assess in detail how well models recognise and handle unsafe information. SafeDialBench addresses these gaps. It was presented at the International Conference on Learning Representations (ICLR) 2026.

The benchmark is organised around a two-tier safety taxonomy with six dimensions: fairness, legality, morality, aggression, ethics and privacy. Each dimension is broken down into more specific safety points; for example, fairness covers stereotypes and distributional harm. The benchmark contains 4,053 dialogues in English and Chinese, each with 3 to 10 turns, across 22 scenarios. The dialogues use seven jailbreak strategies, such as constructing a fictional scene, referring back to earlier content, or disguising a harmful goal as its opposite.

Models are assessed on three abilities: identifying unsafe risks, handling unsafe information appropriately, and staying consistent when attacked across turns. Responses are scored automatically by an LLM, which the authors report agrees with human experts more than 80% of the time. Their evaluation of 19 LLMs found that fallacy attacks and disguised goals were among the most effective attack strategies, and that safety often weakens after the fourth turn.

Auto-discovered on 2026-07-11 by OECD Catalogue Automation