结合标注者态度重新审视性别歧视与厌女症分类
Re-examining Sexism and Misogyny Classification with Annotator Attitudes
AI总结:
本研究针对现有GBV数据集的标注视角局限性,分析标注者身份态度与标注结果的关联,并通过融入标注者信息的提示策略优化大语言模型分类性能,为性别歧视内容审核提供新视角。
摘要:
基于性别的暴力(GBV)在网络上日益严峻,但现有数据集未能捕捉标注者视角的多元性,也无法保障受影响群体的代表性。我们重新审视GBV审核流程中的两个关键阶段:(1)人工数据标注;(2)自动分类。针对(1),我们分析两个数据集,探究标注者身份、态度与其在两项GBV标注任务中作答结果的关联。为此,我们借助社会心理学领域三份经过验证的调查问卷,从众包标注者处收集人口统计信息与态度数据。研究发现,右翼威权主义得分越高,将文本标注为性别歧视的倾向越强;而社会支配取向与新性别歧视态度得分越高,做出该类标注的负面倾向越显著。针对(2),我们使用大语言模型与五种提示策略开展分类实验,包括在提示中融入标注者信息。结果表明:(i)标注者态度会影响分类器预测其标注结果的能力;(ii)当使用结构清晰的简短标注者描述时,加入态度信息可提升模型性能;(iii)模型难以适配新标注集更高的复杂度与类别不平衡问题。
英文摘要:
Gender-Based Violence (GBV) is an increasing problem online, but existing datasets fail to capture the plurality of possible annotator perspectives or ensure the representation of affected groups. We revisit two important stages in the moderation pipeline for GBV: (1) manual data labelling; and (2) automated classification. For (1), we examine two datasets to investigate the relationship between annotator identities and attitudes and the responses they give to two GBV labelling tasks. To this end, we collect demographic and attitudinal information from crowd-sourced annotators using three validated surveys from Social Psychology. We find that higher Right Wing Authoritarianism scores are associated with a higher propensity to label text as sexist, while for Social Dominance Orientation and Neosexist Attitudes, higher scores are associated with a negative tendency to do so. For (2), we conduct classification experiments using Large Language Models and five prompting strategies, including infusing prompts with annotator information. We find: (i) annotator attitudes affect the ability of classifiers to predict their labels; (ii) including attitudinal information can boost performance when we use well-structured brief annotator descriptions; and (iii) models struggle to reflect the increased complexity and imbalanced classes of the new label sets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。