AuditNLG

AuditNLG is an open-source Python library designed to assess and improve the trustworthiness of text generated by large language models (LLMs). It helps reduce risks associated with using generative AI by evaluating outputs across three key dimensions:

1. Factualness – checks whether a text is consistent with provided or external knowledge, ensuring it is free from hallucinations and aligns with factual information.
2. Safety – detects harmful, biased, toxic, or sensitive content, including hate speech, identity attacks, violence, sexual or profane content, and other unsafe language.
3. Constraint adherence – evaluates whether the text follows explicit or implicit instructions, formats, styles, or target audience requirements.

The toolkit integrates multiple evaluation approaches, including LLM-based scoring, QA-based factuality verification, and specialised safety classifiers, providing quantitative scores, structured metadata, and textual explanations for each assessed output. It supports both API-based models (e.g., OpenAI) and local open-source models, allowing flexible deployment depending on infrastructure, privacy, or compliance needs.

AuditNLG also features the PromptHelper module, which leverages self-refinement and rewriting techniques to automatically suggest improved versions of problematic outputs. This enhances factual accuracy, safety, and compliance with constraints.

The system uses structured JSON inputs containing prompts, generated outputs, and optional grounding knowledge. It can be accessed through a Python package or command-line interface, making it suitable for integration into evaluation pipelines, AI research workflows, governance processes, and auditing frameworks.

The toolkit is primarily intended for developers, data scientists, and system operators, while also supporting regulators and policy communities seeking structured, auditable assessments of generative AI outputs. By providing standardized metrics and explanations, AuditNLG supports trustworthy AI evaluation, risk management, and governance practices.

As an open-source resource, AuditNLG encourages contributions from the community to expand its methods, improve accessibility, and enhance the evaluation of generative AI across domains.