Toxicity detection algorithms, originally designed with reactive content moderation in mind, are increasingly being deployed into proactive end-user interventions to moderate content. Through a socio-technical lens and focusing on contexts in which they are applied, we explore the use of these algorithms in proactive moderation systems. Placing a toxicity detection algorithm in an imagined virtual mobile keyboard, we critically explore how such algorithms could be used to proactively reduce the sending of toxic content. We present findings from design workshops conducted with four distinct stakeholder groups and find concerns around how contextual complexities may exasperate inequalities around content moderation processes. Whilst only specific user groups are likely to directly benefit from these interventions, we highlight the potential for other groups to misuse them to circumvent detection, validate and gamify hate, and manipulate algorithmic models to exasperate harm.
A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement
TECNALIA, Basque Research and Technology Alliance (BRTA)(TECNALIA巴斯克研究与技术联盟);University of the Basque Country (UPV/EHU)(巴斯克国家大学);Faculty of Engineering, University of Deusto(德乌斯托大学工程学院)
The Big Ban Theory: A Pre- and Post-Intervention Dataset of Online Content Moderation Actions
IIT-CNR, Italy(意大利国家研究委员会意大利理工学院);Dept. of Computer Science, University of Pisa, Italy(比萨大学计算机系);Dept. of Information Engineering, University of Pisa, Italy(比萨大学信息工程系)