garak

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse and growing. This includes anyone with access to the internet with a decent command of natural language. 

Further, what constitutes a security weakness in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. 

With garak it is time to rethink what constitutes "LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. Introduced in this paper, garak (Generative AI Red-teaming and Assessment Kit), is a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. Garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.

Garak is an open-source tool managed and shared on github.