Auditing a City of Vienna AI solution with a trustworthy AI framework

IEEE SA has been developing a certification process for AI systems called IEEE CertifAIEd, which is based on the work of the IEEE Ethics Certification Program for Autonomous and Intelligent Systems (ECPAIS). IEEE CertifAIEd defines a risk and conformity assessment procedure to follow for a specific AI system or service. The institution seeking certification for its product, service, or system provides detailed insights to a trusted authority according to the IEEE CertifAIEd guidelines. After successful completion the institution receives a certificate, stating that the product meets the ethical criteria set forth by IEEE CertifAIEd.

IEEE CertifAIEd defines four criteria sets for the assessment of ethical properties of autonomous and intelligent systems:

• - Transparency
• - Accountability
• - Algorithmic Bias
• - Privacy

Each of these is of higher or lower relevance to a specific system or service and its application domain. Accordingly, any subset or all of them may be selected for carrying out the assessment of a given product. 

Recently IEEE CertifAIEd’s first real-world test was completed in a pilot project between IEEE SA and Wiener Stadtwerke, by applying IEEE CertifAIEd process to a concrete AI application. Wiener Stadtwerke is a public service provider owned by the City of Vienna, providing services in the areas of public transport, electricity, natural gas, heating, telecommunications, parking, burial, and cemeteries, to more than two million customers in the Vienna metropolitan region. With more than 15,000 employees and annual revenue of more than three billion euros, the Wiener Stadtwerke group belongs to the largest 25 companies in Austria.

As a regional infrastructure service provider, Wiener Stadtwerke makes a significant contribution to the high quality of life in Vienna and is the number one provider of public services to the citizens. Sustainability and Corporate Social Responsibility (CSR) are central and integral parts of the corporate strategy. Vienna’s citizens and the general public justifiably have high expectations towards “their” service provider, including the area of ethics.

In recent years, the Wiener Stadtwerke group has explored several ideas for using AI technology in pilot projects, always adhering to the overall goal of efficiently delivering high-quality services to the citizens of Vienna. One of these was selected for thorough ethical evaluation in the IEEE CertifAIEd pilot with IEEE SA. This is an email classification system (ECS), which is used to automatically assign categories to incoming customer service requests.

The customer service department of Wien Energie (an energy provider belonging to the Wiener Stadtwerke group) receives more than 1,000 email requests per day, which need to be briefly skimmed over by a person and assigned to one of about 15 categories. This categorization results in tickets assigned to different teams for processing, where every email is read by a human operator, who will then determine and initiate the appropriate actions and send a reply to the customer. The manual pre-categorization procedure amounts to one person’s entire work time per day, even when less than 30 seconds are spent per email. And it is a very repetitive, monotonous, and tiring task. The ECS was developed to automate this pre-categorization step, effectively relieving one customer service operator to focus on actual customer interaction again, and thus making better use of their qualifications and training.

Because the described manual procedure has been applied for years, an excellent data collection of several hundred thousand emails with manual category assignments by experts was readily available, providing a very promising starting position for a machine learning approach to the problem. Furthermore, a new data scientist had recently been hired at Wiener Stadtwerke with an AI research background and strong expertise and experience in the field of natural language processing. Therefore, a group-internal project was initiated in 2019 to explore the possibility to develop an automatic categorization system from scratch, which gradually led via increasingly mature prototypes to a production-ready email classification system.

For Wiener Stadtwerke as an organization, the work on this particular application is really only the beginning of a journey towards ethical excellence in the use of AI technology. The expert panel feedback contains many useful pointers to things that could be further improved. The experience gained in this pilot evaluation will make further assessments of new systems or services easier – probably for both Wiener Stadtwerke and IEEE SA.

After undergoing the IEEE CertifAIEd procedure with our email classification system, we believe at Wiener Stadtwerke that IEEE SA has developed an excellent framework for thorough ethical evaluation of intelligent autonomous systems. It is suitable for practice from the point of view of an organization seeking certification for its products, and it exercises sufficient scrutiny to adequately protect the interests of affected individuals and society at large.

This article does not constitute legal or other professional advice and was written by students as part of the Duke Ethical Technology Practicum.