Re-TACRED：解决TACRED数据集的缺陷
Re-TACRED: Addressing Shortcomings of the TACRED Dataset
摘要：
TACRED是规模最大、使用最广泛的关系级关系抽取数据集之一。使用该数据集评估的拟议模型始终设置新的最先进的性能。然而，尽管利用了外部知识和大型文本语料库上的无监督预训练，它们仍然表现出很大的出错率。最近的一项研究表明，这可能是由于数据集质量差。该研究发现，在开发和测试集中，超过50%的最具挑战性的句子被错误标记，导致模型性能平均下降8%。然而，这项研究仅限于5 k（总共106 k）个句子的小样本，大大限制了其发现的普遍性和更广泛的影响。在本文中，我们通过以下方式解决这些缺点：（i）对整个TACRED数据集进行全面研究，（ii）提出改进的众包策略并将其部署为重新注释整个数据集，以及（iii）进行彻底分析以了解纠正TACRED注释如何影响先前发布的结果。经过验证，我们观察到23.9%的TARED标签不正确。此外，在我们修改后的数据集上评估几个模型，平均f1分数提高了14.3%，有助于揭示不同模型之间的重要关系（而不是简单地通过常数因子抵消或缩放它们的分数）。最后，除了我们的分析，我们还发布了Re-TACRED，这是一个新的完全重新注释的TACRED数据集版本，可用于执行关系抽取模型的可靠评估。
英文摘要：
TACRED is one of the largest and most widely used sentence-level relation extraction datasets. Proposed models that are evaluated using this dataset consistently set new state-of-the-art performance. However, they still exhibit large error rates despite leveraging external knowledge and unsupervised pretraining on large text corpora. A recent study suggested that this may be due to poor dataset quality. The study observed that over 50% of the most challenging sentences from the development and test sets are incorrectly labeled and account for an average drop of 8% f1-score in model performance. However, this study was limited to a small biased sample of 5k (out of a total of 106k) sentences, substantially restricting the generalizability and broader implications of its findings. In this paper, we address these shortcomings by: (i) performing a comprehensive study over the whole TACRED dataset, (ii) proposing an improved crowdsourcing strategy and deploying it to re-annotate the whole dataset, and (iii) performing a thorough analysis to understand how correcting the TACRED annotations affects previously published results. After verification, we observed that 23.9% of TACRED labels are incorrect. Moreover, evaluating several models on our revised dataset yields an average f1-score improvement of 14.3% and helps uncover significant relationships between the different models (rather than simply offsetting or scaling their scores by a constant factor). Finally, aside from our analysis we also release Re-TACRED, a new completely re-annotated version of the TACRED dataset that can be used to perform reliable evaluation of relation extraction models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
