Model Card Regulatory Check

The Model Card Regulatory Check is an app devoted to automating regulatory compliance of AI systems based on accepted documentation tools in the AI space, such as model cards. This app is an example of how different governance tools can be articulated in order to bring a more efficient AI governance system. Despite AI governance tools' (e.g. regulation, model cards, licenses or ethical charters) intrinsic value, the former might become more valuable when articulated with each other (i.e. a systemic approach to AI governance). 

This app was designed as a pilot case and specifically checks whether the information available in a model card matches with certain regulatory requirements of article 13 of the proposed EU AI Act (more specifically, the General Approach of the Council of 25th November 2022). Article 13 targets transparency and provision of information to users such as the intended purpose of the AI system, its limitations, or computational specifications. Interestingly, a complete model card could provide all this information and thus be used as a regulatory compliance tool, beyond being a documentation tool for users.

The app is built with two components: a modular Python library that implements individual “checks”, and a user interface built with Gradio and the Hugging Face Hub API. The Python library is designed to parse Markdown files that are expected to match Hugging Face’s “model card” specification. Each check scans the provided file to verify that specific content headings are present. The user interface allows users to search for existing model cards on the Hugging Face Hub, or upload their own Markdown file. Once the card is loaded, the tool runs the checks and displays the results to the user.

Compliance with the EU AI Act for high-risk AI systems is estimated in tens of thousands of euros (if not hundreds). SMEs developing high-risk AI systems, especially startups, might not have the time and financial resources needed to implement compliance and certify their AI systems. Henceforth, tools like the Model Card Regulatory Check can contribute to lower the cost of regulatory compliance by bringing simple and quick checks to both the developers of the AI system and the agencies or authorities willing to inspect it. From a different angle, the articulation between model cards and regulation might incentivize AI developers to devote more time to documentation and model cards, as the information provided will serve both the users and authorities.