arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.10629cs.HC
收藏

对毒性检测算法在主动内容审核系统中应用的批判性反思

A Critical Reflection on the Use of Toxicity Detection Algorithms in Proactive Content Moderation Systems

Mark Warner, Angelika Strohmayer, Matthew Higgs, Lynne Coventry

AI总结:

本研究从社会技术视角出发,通过设计工作坊批判性反思毒性检测算法在主动内容审核系统中的应用,发现语境复杂性或加剧审核不平等,且存在被滥用加剧伤害的风险。

摘要:

毒性检测算法最初是为响应式内容审核设计的,如今正越来越多地被部署到面向终端用户的主动干预措施中以审核内容。我们从社会技术视角出发,聚焦其应用场景,探究这类算法在主动审核系统中的使用情况。我们将一款毒性检测算法置于设想的虚拟移动键盘中,批判性地探讨了此类算法可如何被用于主动减少有毒内容的发送。我们展示了与四个不同利益相关者群体开展的设计工作坊的研究结果,发现人们担忧语境复杂性可能会加剧内容审核流程中的不平等问题。虽然只有特定用户群体可能直接从这些干预措施中受益,但我们也指出了其他群体滥用这些措施的可能性:他们可能借此规避检测、将仇恨言论正当化并将其游戏化,还可能操纵算法模型以加剧伤害。

英文摘要:

Toxicity detection algorithms, originally designed with reactive content moderation in mind, are increasingly being deployed into proactive end-user interventions to moderate content. Through a socio-technical lens and focusing on contexts in which they are applied, we explore the use of these algorithms in proactive moderation systems. Placing a toxicity detection algorithm in an imagined virtual mobile keyboard, we critically explore how such algorithms could be used to proactively reduce the sending of toxic content. We present findings from design workshops conducted with four distinct stakeholder groups and find concerns around how contextual complexities may exasperate inequalities around content moderation processes. Whilst only specific user groups are likely to directly benefit from these interventions, we highlight the potential for other groups to misuse them to circumvent detection, validate and gamify hate, and manipulate algorithmic models to exasperate harm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑