Stability

Robustness Metrics provides lightweight modules in order to evaluate the robustness of classification models. Stability is defined as, e.g. the stability of the prediction and predicted probabilities under natural perturbation of the input.

The library includes popular out-of-distribution datasets (ImageNetV2, ImageNet-C, etc.) and can be readily applied to benchmark arbitrary models and is not limited to vision models: any mapping from input -> logits will do.

Trustworthy AI Relevance

This metric addresses Robustness and Human Agency & Control by quantifying relevant system properties. Robustness: Stability directly measures a system's ability to maintain consistent performance under distribution shifts, noisy inputs, model retraining, or adversarial perturbations (examples: jitter metrics, output variance under small input changes, consistency across continuous data updates). Measuring stability detects brittleness and regression, informs defenses (robust training, input sanitization, OOD detection), and is therefore a core robustness indicator. Human Agency & Control: Stable, predictable model behavior supports user autonomy and control by making outputs more interpretable and less surprising.