These tools and metrics are designed to help AI actors develop and use trustworthy AI systems and applications that respect human rights and are fair, transparent, explainable, robust, secure and safe.
Uncovering bugs in a health care model

AI models are valuable when they are accurate. While model performance is important in any context, accuracy becomes especially critical in high-risk contexts like health care.
Google’s What-If Tool assesses a variety of model metrics, including performance. When a model performs well, it means its predictions or decisions are more likely to be correct. A software engineer at a large technology company was able to use the tool to understand the performance of two different versions of a health metric prediction model, giving them the information to make improvements.
The two models were trained with different data. The first model used the initial dataset while the second model used a newer one that used different input features generated by feature-building code. The tool found that the first model’s predictions for the health metric were almost always larger than those of the second model. It also found the first model’s training data was almost always larger than the actual initial health metric measurement. This wasn’t the case for the second model’s data.
This observation prompted an investigation that found a bug in the code that swapped measurement values for different input features in the first model’s dataset. The engineer said that before using the What-If Tool, their team didn’t spot or suspect any bugs in the code.
Benefits of using the tool in this use case
While in this particular use case the user knew how to code, someone without this skillset could have performed the same performance analysis using the What-If Tool. This democratization of AI performance evaluation enables a greater share of an organization’s employees to play an active role.
Shortcomings of using the tool in this use case
The What-If Tool revealed performance issues of one of the models, but the software engineer did not use the tool to diagnose the bug in the code. That last step still required knowledge of how to navigate the codebase without the assistance of the What-If Tool.
Learnings or advice for using the tool in a similar context
When using the What-If Tool to evaluate the performance of models used in high-risk situations like health care, also take advantage of the tool’s fairness analysis features.
This article does not constitute legal or other professional advice and was written by students as part of the Duke Ethical Technology Practicum.
About the use case
You can click on the links to see the associated use cases
Target sector(s):
Country/Territory of origin:


























