一致性是关键:利用标注者内部一致性协议厘清自然语言处理中的标签差异
Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement
AI总结:
本文主张在NLP标注中引入标注者内部一致性以衡量标签稳定性,并通过系统综述与四项探索性实验发现标注者约25%的回答存在不一致。
摘要:
在自然语言处理(NLP)任务中,我们通常使用一致性度量来评估人类标注者所作判断的效用。尽管标注者间一致性常通过衡量不同标注者之间的一致性而被用作标签可靠性的指标,但我们主张还应使用标注者内部一致性,以衡量标签随时间变化的稳定性(以及标注者的一致性)。然而,在一项系统性综述中,我们发现该领域很少报告后者。计算这些度量可作为重要的质量控制手段,并有助于揭示标注者产生分歧的原因。我们开展了探索性标注实验,以研究这些度量与对文本项目主观性和模糊性的感知之间的关系,结果发现,在四项不同的NLP任务中,标注者约有25%的时间会给出不一致的回答。
英文摘要:
We commonly use agreement measures to assess the utility of judgements made by human annotators in Natural Language Processing (NLP) tasks. While inter-annotator agreement is frequently used as an indication of label reliability by measuring consistency between annotators, we argue for the additional use of intra-annotator agreement to measure label stability (and annotator consistency) over time. However, in a systematic review, we find that the latter is rarely reported in this field. Calculating these measures can act as important quality control and could provide insights into why annotators disagree. We conduct exploratory annotation experiments to investigate the relationships between these measures and perceptions of subjectivity and ambiguity in text items, finding that annotators provide inconsistent responses around 25% of the time across four different NLP tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。