Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of Explanations
Faculty of Computer Science, UniVie(维也纳大学计算机科学系);Doctoral School Computer Science, Faculty of Philological and Cultural Studies, University of Vienna, Austria(维也纳大学哲学与文化研究系计算机科学博士学院);MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(慕尼黑大学信息与语言处理中心MaiNLP);Munich Center for Machine Learning, Germany(慕尼黑机器学习中心);FNRS, CENTAL, UCLouvain, Belgium(比利时列日大学FNRS、CENTAL)
Pingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe, Benjamin Roth, Barbara Plank
备注:
Accepted by ACL 2026 Findings, 13 pages, 6 figures
Natural Language Inference (NLI) datasets often exhibit human label variation. To better understand these variations, explanation-based approaches analyze the underlying reasoning behind annotators' decisions. One such approach is the LiTEx taxonomy, which categorizes free-text explanations in English into reasoning categories. However, previous work applying LiTEx has focused on within-label variation: cases where annotators agree on the NLI label but provide different explanations. This paper broadens the scope by examining how annotators may diverge not only in the reasoning category but also in the labeling. We use explanations as a lens to analyze variation in NLI annotations and to examine individual differences in reasoning. We apply LiTEx to two NLI datasets and align annotation variation from multiple aspects: NLI label agreement, explanation similarity, and taxonomy agreement, with an additional compounding factor of annotators' selection bias. We observe instances where annotators disagree on the label but provide similar explanations, suggesting that surface-level disagreement may mask underlying agreement in interpretation. Moreover, our analysis reveals individual preferences in explanation strategies and label choices. These findings highlight that agreement in reasoning categories better reflects the semantic similarity of explanations than label agreement alone. Our findings underscore the richness of reasoning-based explanations and the need for caution in treating labels as ground truth.
Designing NLP Systems That Adapt to Diverse Worldviews
Faculty of Mathematics and Computer Science(数学与计算机科学学院);Interdisciplinary School of Doctoral Studies(跨学科博士研究学院);HLT Research Center(HLT研究中心);University of Bucharest(布加勒斯特大学)
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany(MaiNLP,信息与语言处理中心,慕尼黑大学,德国);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML),慕尼黑,德国);Language Technology Lab, University of Cambridge, United Kingdom(语言技术实验室,剑桥大学,英国)