arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.16458cs.CL
收藏

一致、分歧、解释:通过解释的视角分解自然语言推理中的人类标注变异

Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations

Faculty of Computer Science, UniVie(维也纳大学计算机科学系);Doctoral School Computer Science, Faculty of Philological and Cultural Studies, University of Vienna, Austria(维也纳大学哲学与文化研究系计算机科学博士学院);MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(慕尼黑大学信息与语言处理中心MaiNLP);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);FNRS, CENTAL, UCLouvain, Belgium(比利时列日大学FNRS、CENTAL)

Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth, Barbara Plank

AI总结:

本文通过分析解释视角,探讨自然语言推理中人类标注变异的多样性,揭示标注者在推理类别和标签上的分歧,强调推理类别一致性比标签一致性更能反映解释的语义相似性。

摘要:

自然语言推理(NLI)数据集往往表现出人类标注的变异。为了更好地理解这些变异,基于解释的方法分析标注者决策背后的推理过程。其中一种方法是LiTEx分类法,将英文自由文本解释分为推理类别。然而,此前应用LiTEx的研究主要集中在同标签内变异:标注者在NLI标签上一致但提供不同解释的情况。本文扩大了研究范围,考察标注者在推理类别和标签上可能的分歧。我们利用解释作为分析NLI标注变异的视角,并考察个体在推理上的差异。我们对两个NLI数据集应用LiTEx,并从多个方面对标注变异进行对齐:NLI标签一致性、解释相似性和分类一致性,同时增加标注者选择偏见的附加因素。我们观察到标注者在标签上分歧但提供相似解释的情况,表明表面分歧可能掩盖了解释层面的一致性。此外,我们的分析揭示了个体在解释策略和标签选择上的偏好。这些发现强调,推理类别的共识比单纯的标签一致更能反映解释的语义相似性。我们的发现突显了基于推理的解释的丰富性,并强调在将标签视为真实值时需保持谨慎。

英文摘要:

Natural Language Inference (NLI) datasets often exhibit human label variation. To better understand these variations, explanation-based approaches analyze the underlying reasoning behind annotators' decisions. One such approach is the LiTEx taxonomy, which categorizes free-text explanations in English into reasoning categories. However, previous work applying LiTEx has focused on within-label variation: cases where annotators agree on the NLI label but provide different explanations. This paper broadens the scope by examining how annotators may diverge not only in the reasoning category but also in the labeling. We use explanations as a lens to analyze variation in NLI annotations and to examine individual differences in reasoning. We apply LiTEx to two NLI datasets and align annotation variation from multiple aspects: NLI label agreement, explanation similarity, and taxonomy agreement, with an additional compounding factor of annotators' selection bias. We observe instances where annotators disagree on the label but provide similar explanations, suggesting that surface-level disagreement may mask underlying agreement in interpretation. Moreover, our analysis reveals individual preferences in explanation strategies and label choices. These findings highlight that agreement in reasoning categories better reflects the semantic similarity of explanations than label agreement alone. Our findings underscore the richness of reasoning-based explanations and the need for caution in treating labels as ground truth.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑