Eticas Bias

This framework is designed to evaluate AI systems comprehensively across all lifecycle stages. At its core, it compares privileged and underprivileged groups, ensuring a fair evaluation of model behavior.

This framework, with its wide range of metrics focuses on bias monitoring. It offers a profound perspective on fairness, allowing for comprehensive reporting even without relying on true labels. The only restriction on measuring bias in production is performance metrics, as they are directly tied to rue labels.

The stages considered are the following:

1. The dataset used to train the model.
2. The dataset used in production.
3. A dataset containing the system’s final decisions, which may include human intervention or another model.

• Demographic Benchmarking Monitoring: Perform in-depth analysis of population distribution.
• Model Fairness Monitoring: Ensure equality and detect equity issues in decision-making.
• Features Distribution Evaluation: Analyze correlations, causality, and variable importance.
• Performance Analysis: Metrics to assess model performance, accuracy, and recall.
• Model Drift Monitoring: Detect and measure changes in data distributions and model behavior over time.