Use cases using the IBM Factsheets

There are several examples of use cases developed by IBM on how Factsheets can be built in practice. One example relates to the audio classifier model, which classifies an input audio clip into five different classes it detects. A first version of the related FactSheet was developed based on interviews with the data scientists who built the model at IBM. This helped identify the possible information about the model that an AI developer/customer who wishes to use it would need. This first version was then further refined through several iterations with field experts and possible users. The whole process represented approximately one week of effort.

Another example is a mortgage evaluator governance model that predicts mortgage approval. The FactSheet for this model was the result of both an iterative interview-based process as noted above, regarding for example free form questions such as the purpose of the model or the required tests for the model (“model policy”), as well as an automated process developed by IBM. The latter was used to capture information about the work of the data scientist (such as possible data manipulations, definitions of new features, or tests performed) while building the model. Ultimately such an automated process can greatly facilitate the development of FactSheets in accordance with corporate level “model policies” (e.g., information requirements regarding the accuracy, fairness, explainability, or robustness of the AI system).

Read full use case here.