TextEE:事件抽取中的基准、重新评估、反思与未来挑战
TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction
AI总结:
本文提出标准化事件抽取基准TextEE,覆盖16个数据集和14种方法,通过重新评估揭示现有评估问题及大语言模型的不足,并为未来研究提供挑战与见解。
摘要:
事件抽取因其广泛的应用而引起了相当大的关注。然而,近期研究指出了评估方面的问题,表明所报告的分数可能无法准确反映真实性能。在本工作中,我们识别并应对评估挑战,包括由于不同的数据假设或预处理步骤所导致的不一致性、当前评估框架可能引入数据集或数据划分偏差的不足,以及一些先前方法的低可复现性。为应对这些挑战,我们提出了TextEE,一个标准化、公平且可复现的事件抽取基准。TextEE包含针对16个数据集(涵盖八个不同领域)的标准化数据预处理脚本和数据划分,并纳入14种近期方法,进行了全面的基准重新评估。我们还在TextEE基准上评估了五种不同的大语言模型,并展示了它们难以取得令人满意的性能。受我们的重新评估结果和发现的启发,我们讨论了事件抽取在当前自然语言处理时代中的作用,以及从TextEE中得出的未来挑战和见解。我们相信,TextEE作为首个标准化综合基准工具,将极大地促进未来的事件抽取研究。
英文摘要:
Event extraction has gained considerable interest due to its wide-ranging applications. However, recent studies draw attention to evaluation issues, suggesting that reported scores may not accurately reflect the true performance. In this work, we identify and address evaluation challenges, including inconsistency due to varying data assumptions or preprocessing steps, the insufficiency of current evaluation frameworks that may introduce dataset or data split bias, and the low reproducibility of some previous approaches. To address these challenges, we present TextEE, a standardized, fair, and reproducible benchmark for event extraction. TextEE comprises standardized data preprocessing scripts and splits for 16 datasets spanning eight diverse domains and includes 14 recent methodologies, conducting a comprehensive benchmark reevaluation. We also evaluate five varied large language models on our TextEE benchmark and demonstrate how they struggle to achieve satisfactory performance. Inspired by our reevaluation results and findings, we discuss the role of event extraction in the current NLP era, as well as future challenges and insights derived from TextEE. We believe TextEE, the first standardized comprehensive benchmarking tool, will significantly facilitate future event extraction research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。