AILuminate benchmark v1.0

The AILuminate v1.0 benchmark, developed by MLCommons, evaluates the safety of AI systems by testing how they respond to a set of prompts related to potential harms.

The benchmark can be applied to both bare models, which are standalone models without external guardrails, and AI systems, which may include additional components such as moderation filters, guardrails, or other safety mechanisms. To evaluate a system, AILuminate inputs prompts into the system under test (SUT), records its responses, and analyzes them using specialized safety evaluator models. These evaluator models determine whether the responses violate the AILuminate Assessment Standard guidelines. The benchmark examines several categories of hazards, including physical harms (such as violent crime or self-harm), non-physical harms (such as hate speech, privacy violations, or intellectual property issues), and contextual hazards like unsafe specialized advice. The findings are summarized in a human-readable report, and each system receives a grade ranging from Poor to Excellent based on the percentage of responses that violate the assessment standard. This allows different AI systems to be compared according to their observed safety performance.