VariErr NLI：区分标注错误与人类标签变异
VariErr NLI: Separating Annotation Error from Human Label Variation
AI总结：
本文提出 VariErr NLI 数据集和两轮标注方法，用于区分标注错误与人类标签变异，并发现现有自动错误检测方法及 GPT-4 仍不及人类表现。
摘要：
当标注者出于正当理由对同一项目赋予不同标签时，会产生人类标签变异；而当标签是出于不正当理由被赋予时，则属于标注错误。这两个问题在自然语言处理（NLP）基准中普遍存在，然而现有研究一直将它们孤立地加以研究。据我们所知，此前尚无专门致力于将错误与信号区分开来的工作，尤其是在信号并非非黑即白的情况下。为填补这一空白，我们引入了一套系统化方法以及一个新数据集 VariErr（variation versus error，变异与错误），重点关注英语自然语言推理（NLI）任务。我们提出了一种两轮标注流程：标注者先解释每个标签，随后判断标签-解释对的有效性。VariErr 包含针对 500 个重新标注的 MNLI 项目所产生的 1,933 条解释的 7,732 条有效性判断。我们评估了各种自动错误检测（AED）方法以及 GPT 模型在区分错误与人类标签变异方面的有效性。我们发现，最先进的 AED 方法显著落后于 GPT 和人类。尽管 GPT-4 是最佳系统，但仍未达到人类表现。我们的方法可应用于 NLI 之外的领域，为未来研究错误与合理变异之间的区别提供了广阔空间，进而有助于构建更优、更可信的 NLP 系统。
英文摘要：
Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons. These two issues are prevalent in NLP benchmarks, yet existing research has studied them in isolation. To the best of our knowledge, there exists no prior work that focuses on teasing apart error from signal, especially in cases where signal is beyond black-and-white. To fill this gap, we introduce a systematic methodology and a new dataset, VariErr (variation versus error), focusing on the NLI task in English. We propose a 2-round annotation procedure with annotators explaining each label and subsequently judging the validity of label-explanation pairs. VariErr contains 7,732 validity judgments on 1,933 explanations for 500 re-annotated MNLI items. We assess the effectiveness of various automatic error detection (AED) methods and GPTs in uncovering errors versus human label variation. We find that state-of-the-art AED methods significantly underperform GPTs and humans. While GPT-4 is the best system, it still falls short of human performance. Our methodology is applicable beyond NLI, offering fertile ground for future research on error versus plausible variation, which in turn can yield better and more trustworthy NLP systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
