It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators. Doing so can remove valuable nuances and diverse perspectives inherent in the annotators' assessments. This paper proposes and compares three different strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning. All three approaches are evaluated on the tasks of hate speech and abusive conversation detection, which inherently entail a high degree of subjectivity. Moreover, to evaluate the effectiveness of embracing annotation disagreements for model training, we conduct an online survey that compares the performance of the multi-label model against a baseline model, which is trained with the majority label.
The results show that in hate speech detection, the multi-label method outperforms the other two approaches, while in abusive conversation detection, instruction tuning achieves the best performance. The results of the survey also show that the outputs from the multi-label models are considered a better representation of the texts than the single-label model.
越来越多的证据表明,用于构建自然语言推理数据集的原始注释中普遍存在的分歧使得将这些注释聚合到单个标签的常见做法存在问题。我们提出了一种通用的方法,允许人们跳过聚合步骤,直接在原始注释上进行训练,而不会使模型受到注释者响应偏差引起的不必要的噪声。我们证明,这种方法,它概括了\textit{mixed effects model}的概念,将\textit{annotator random effects}到任何现有的神经模型,提高了性能的模型,不包括这样的效果。
SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)
Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会);Heriot-Watt University(赫瑞瓦特大学);Queen Mary University of London(伦敦大学玛丽皇后学院);Università di Torino(都灵大学);Università Bocconi(博科尼大学);University of Munich(慕尼黑大学)
Designing NLP Systems That Adapt to Diverse Worldviews
Faculty of Mathematics and Computer Science(数学与计算机科学学院);Interdisciplinary School of Doctoral Studies(跨学科博士研究学院);HLT Research Center(HLT研究中心);University of Bucharest(布加勒斯特大学)
A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement
Effi Levi Department of Computer Science, The Hebrew University of Jerusalem(Effi Levi 计算机科学系,耶路撒冷希伯来大学);Shaul R. Shenhav Department of Political Science, The Hebrew University of Jerusalem(Shaul R. Shenhav 政治学系,耶路撒冷希伯来大学)
Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations
Faculty of Computer Science, UniVie(维也纳大学计算机科学系);Doctoral School Computer Science, Faculty of Philological and Cultural Studies, University of Vienna, Austria(维也纳大学哲学与文化研究系计算机科学博士学院);MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(慕尼黑大学信息与语言处理中心MaiNLP);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);FNRS, CENTAL, UCLouvain, Belgium(比利时列日大学FNRS、CENTAL)
Investigating Disagreement in the Scientific Literature
Centre for Science and Technology Studies, Leiden University(莱顿大学科学技术研究中心);SciTech Strategies, Inc.(SciTech Strategies 公司);School of Public Policy, Georgia Institute of Technology(佐治亚理工学院公共政策学院);School of Informatics, Computing, and Engineering, Indiana University Bloomington(印第安纳大学伯明顿分校信息学、计算与工程学院)
Diverging Preferences: When do Annotators Disagree and do Models Know?
New York University(纽约大学);Allen Institute for Artificial Intelligence(人工智能研究院);NVIDIA(NVIDIA公司);University of Washington(华盛顿大学);University of Southern California(南加州大学)
Dealing with Annotator Disagreement in Hate Speech Classification
Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey(工程与自然科学学院,Sabanci大学,伊斯坦布尔,土耳其);Center of Excellence in Data Analytics (VERIM), Sabanci University, Istanbul, Turkey(数据分析卓越中心(VERIM),Sabanci大学,伊斯坦布尔,土耳其)
Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks
Instituto Politécnico Nacional(智利国家理工学院);University of Wisconsin–Stevens Point(威斯康星州立大学斯蒂文斯点分校);University of Pretoria(培特里亚大学);Bahir Dar University(巴希尔达大学);Imperial College London(伦敦帝国理工学院);University of Hamburg(汉堡大学)
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University(生物与社会技术系统建模实验室,东北大学);Heriot-Watt University(赫瑞-沃顿大学);aequa-tech;Università del Piemonte Orientale(皮埃蒙特东方大学);Università degli Studi di Torino(托斯卡纳大学)
The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels
University of California Berkeley(加州大学伯克利分校);Microsoft Research Montréal(微软蒙特利尔研究院);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Human Label Variation as Stable Signal: Learning Annotator-Specific Explanation Behavior via Cross-Annotator Preference Optimization
MaiNLP;Center for Information and Language Processing(信息与语言处理中心);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Vienna(维也纳大学);LTL;University of Cambridge(剑桥大学)