超越精确匹配:对大型语言模型事件抽取的语义重评估
Beyond Exact Match: Semantically Reassessing Event Extraction by Large Language Models
AI总结:
针对词元级精确匹配评估误判语义正确案例并低估模型真实性能的问题,提出基于LLM的语义级事件抽取评估框架RAEE,实现对触发词和论元精确率与召回率的自适应评估,揭示了精确匹配造成的显著性能低估。
摘要:
由于应用广泛,事件抽取获得了广泛的研究关注。然而,当前主流的事件抽取评估方法依赖于词元级精确匹配,这会误判大量语义级正确的案例。这种依赖导致模型在精确匹配标准下的评估性能与其真实性能之间存在显著差异。为了解决这个问题,我们提出了一种可靠且语义化的事件抽取评估框架,名为 RAEE,它在语义级而非词元级准确评估抽取结果。具体而言,RAEE 利用大型语言模型(LLMs)作为评估代理,并结合自适应机制,对触发词和论元的精确率和召回率实现自适应评估。大量实验表明:(1)RAEE 与人类判断实现了非常强的相关性;(2)在 10 个数据集上对包括先进 LLMs 在内的 14 个模型进行重评估后,精确匹配与 RAEE 之间存在显著的性能差距。精确匹配评估显著低估了现有事件抽取模型的性能,特别是低估了 LLMs 的能力;(3)在 RAEE 评估下的细粒度分析揭示了值得进一步探索的深刻现象。我们提出的 RAEE 评估工具包已公开发布。
英文摘要:
Event extraction has gained extensive research attention due to its broad range of applications. However, the current mainstream evaluation method for event extraction relies on token-level exact match, which misjudges numerous semantic-level correct cases. This reliance leads to a significant discrepancy between the evaluated performance of models under exact match criteria and their real performance. To address this problem, we propose a reliable and semantic evaluation framework for event extraction, named RAEE, which accurately assesses extraction results at semantic-level instead of token-level. Specifically, RAEE leverages large language models (LLMs) as evaluation agents, incorporating an adaptive mechanism to achieve adaptive evaluations for precision and recall of triggers and arguments. Extensive experiments demonstrate that: (1) RAEE achieves a very strong correlation with human judgments; (2) after reassessing 14 models, including advanced LLMs, on 10 datasets, there is a significant performance gap between exact match and RAEE. The exact match evaluation significantly underestimates the performance of existing event extraction models, and in particular underestimates the capabilities of LLMs; (3) fine-grained analysis under RAEE evaluation reveals insightful phenomena worth further exploration. The evaluation toolkit of our proposed RAEE is publicly released.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。