arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.17577cs.CL
收藏

利用标注者分歧进行文本分类

Leveraging Annotator Disagreement for Text Classification

University of Twente(特温特大学);Human-Media Interaction(人机交互)

Jin Xu, Mariët Theune, Daniel Braun

AI总结:

本文提出并比较了三种利用标注者分歧进行文本分类的策略,发现在仇恨言论检测中多标签方法表现最佳,在辱骂性对话检测中指令微调最优,且多标签模型输出更能代表文本。

摘要:

在文本分类中,即使数据集由多名标注者标注,通常也只使用一个多数标签进行模型训练。这样做会去除标注者评估中固有的宝贵细微差别和多元视角。本文提出并比较了三种利用标注者分歧进行文本分类的策略:一种基于概率的多标签方法、一个集成系统和指令微调。这三种方法均在本质上具有高度主观性的仇恨言论和辱骂性对话检测任务上进行了评估。此外,为了评估拥抱标注分歧进行模型训练的有效性,我们进行了一项在线调查,比较多标签模型与使用多数标签训练的基线模型的性能。结果显示,在仇恨言论检测中,多标签方法优于其他两种方法,而在辱骂性对话检测中,指令微调取得了最佳性能。调查结果还显示,多标签模型的输出被认为比单标签模型更好地表示了文本。

英文摘要:

It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators. Doing so can remove valuable nuances and diverse perspectives inherent in the annotators' assessments. This paper proposes and compares three different strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning. All three approaches are evaluated on the tasks of hate speech and abusive conversation detection, which inherently entail a high degree of subjectivity. Moreover, to evaluate the effectiveness of embracing annotation disagreements for model training, we conduct an online survey that compares the performance of the multi-label model against a baseline model, which is trained with the majority label. The results show that in hate speech detection, the multi-label method outperforms the other two approaches, while in abusive conversation detection, instruction tuning achieves the best performance. The results of the survey also show that the outputs from the multi-label models are considered a better representation of the texts than the single-label model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑