Crowdsourcing the curation of the training set for harmful content classifiers used in social media

The world’s dominant social media platforms all run active content moderation programmes. They take seriously their responsibility to moderate the content of their users' posts, to keep their community of users safe. Moderation involves checking for ‘harmful content’ of various kinds, and taking various actions when it is found. Some content is removed if it is found to be illegal or if it violates the company’s published standards. Other content is left in place, but is moderated in less draconian ways, perhaps by flagging it with messages advising caution, or by downranking it in the platform’s recommender algorithm. This latter type of content is often termed borderline content, and has been the subject of much discussion. The scale of social media platforms means that content moderation processes must use automated tools, as well as human effort. AI tools are central in moderation processes, so content moderation is an important topic for our group at GPAI, which focusses on Social Media Governance.