GenRES：在大语言模型时代重新思考生成式关系抽取的评估
GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models
AI总结：
本文提出GenRES多维度评估框架，解决传统精确率和召回率无法准确评估生成式关系抽取（GRE）的问题，并在多层级数据集上对十四个领先大模型进行全面基准测试。
摘要：
关系抽取（RE）领域正经历向生成式关系抽取（GRE）的显著转变，这得益于大语言模型（LLMs）的能力。然而，我们发现传统的RE指标（如精确率和召回率）在评估GRE方法时存在不足。这种不足是因为这些指标依赖于与人工标注的参考关系进行精确匹配，而GRE方法通常会生成与参考关系不同但多样且语义准确的关系。为了填补这一空白，我们引入了GenRES，从GRE结果的主题相似性、唯一性、粒度、事实性和完整性方面进行多维度评估。借助GenRES，我们通过实证发现：（1）精确率/召回率无法有效证明GRE方法的性能；（2）人工标注的参考关系可能不完整；（3）使用固定关系或实体集合提示LLMs会导致幻觉。接着，我们对GRE方法进行了人工评估，结果表明GenRES与人类对RE质量的偏好一致。最后，我们分别在文档级、包级和句子级RE数据集上使用GenRES对十四个领先的LLMs进行了全面评估，为未来的GRE研究设定了基准。
英文摘要：
The field of relation extraction (RE) is experiencing a notable shift towards generative relation extraction (GRE), leveraging the capabilities of large language models (LLMs). However, we discovered that traditional relation extraction (RE) metrics like precision and recall fall short in evaluating GRE methods. This shortfall arises because these metrics rely on exact matching with human-annotated reference relations, while GRE methods often produce diverse and semantically accurate relations that differ from the references. To fill this gap, we introduce GenRES for a multi-dimensional assessment in terms of the topic similarity, uniqueness, granularity, factualness, and completeness of the GRE results. With GenRES, we empirically identified that (1) precision/recall fails to justify the performance of GRE methods; (2) human-annotated referential relations can be incomplete; (3) prompting LLMs with a fixed set of relations or entities can cause hallucinations. Next, we conducted a human evaluation of GRE methods that shows GenRES is consistent with human preferences for RE quality. Last, we made a comprehensive evaluation of fourteen leading LLMs using GenRES across document, bag, and sentence level RE datasets, respectively, to set the benchmark for future research in GRE
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
