In NLP annotation, it is common to have multiple annotators label the text and then obtain the ground truth labels based on the agreement of major annotators. However, annotators are individuals with different backgrounds, and minors' opinions should not be simply ignored. As annotation tasks become subjective and topics are controversial in modern NLP tasks, we need NLP systems that can represent people's diverse voices on subjective matters and predict the level of diversity. This paper examines whether the text of the task and annotators' demographic background information can be used to estimate the level of disagreement among annotators. Particularly, we extract disagreement labels from the annotators' voting histories in the five subjective datasets, and then fine-tune language models to predict annotators' disagreement. Our results show that knowing annotators' demographic information, like gender, ethnicity, and education level, helps predict disagreements. In order to distinguish the disagreement from the inherent controversy from text content and the disagreement in the annotators' different perspectives, we simulate everyone's voices with different combinations of annotators' artificial demographics and examine its variance of the finetuned disagreement predictor. Our paper aims to improve the annotation process for more efficient and inclusive NLP systems through a novel disagreement prediction mechanism. Our code and dataset are publicly available.
Designing NLP Systems That Adapt to Diverse Worldviews
Faculty of Mathematics and Computer Science（数学与计算机科学学院）；Interdisciplinary School of Doctoral Studies（跨学科博士研究学院）；HLT Research Center（HLT研究中心）；University of Bucharest（布加勒斯特大学）
The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction
Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University（生物与社会技术系统建模实验室，东北大学）；Heriot-Watt University（赫瑞-沃顿大学）；aequa-tech；Università del Piemonte Orientale（皮埃蒙特东方大学）；Università degli Studi di Torino（托斯卡纳大学）
