Inspect

Inspect is an open-source framework for large language model evaluations created by the UK AI Security Institute. Inspect is a flexible evaluation framework designed to measure model performance across diverse domains, including coding, reasoning, agentic tasks, knowledge, behavior, and multi-modal understanding.

Some of the key features include:

• A set of straightforward interfaces for implementing evaluations and re-using components across evaluations.
• Extensive tooling, including a web-based Inspect View tool for monitoring and visualizing evaluations and a VS Code Extension that assists with authoring and debugging.
• Flexible support for tool calling—custom and MCP tools, as well as built-in bash, python, text editing, web search, web browsing, and computer tools.
• Support for agent evaluations, including flexible built-in agents, multi-agent primitives, the ability to run arbitrary external agents, and agent observability in Inspect View.
• A sandboxing system that supports running untrusted model code in Docker, Kubernetes, Proxmox, and other systems via an extension API.

The Inspect tutorial provides a step-by-step walkthrough of examples of Inspect evaluations.

AI Security Institute, UK. 2024. “Inspect AI: Framework for Large Language Model Evaluations.” https://github.com/UKGovernmentBEIS/inspect_ai.