arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.04969cs.CL
收藏

EVA-Score:通过提取与验证评估抽象长形式摘要的信息性

EVA-Score: Evaluating Abstractive Long-form Summarization on Informativeness through Extraction and Validation

Shanghai Jiao Tong University(上海交通大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);Shanghai AI Lab(上海人工智能实验室)

Yuchen Fan, Yazhe Wan, Xin Zhong, Haonan Cheng, Ning Ding, Bowen Zhou

AI总结:

本文提出EVA-Score,通过提取和验证评估抽象长形式摘要的信息性,发现其与人类评价的关联性最高,揭示了LLM在长形式摘要任务中的不足。

摘要:

自从大型语言模型出现以来,越来越多的关注被投入到抽象长形式摘要中,其中更长的输入序列表示包含更多的信息。然而,此类摘要的自动评估仍然鲜为人知。当前的长形式摘要评估指标要么使用基于相似度的指标如ROUGE和BERTScore,要么使用基于LLM的指标通过适当的提示或预定义的模式。我们认为前者仅依赖于相似度,未能考虑信息性,而后者缺乏对信息丰富度的定量分析,且主观性较强,难以解释。当前的评估指标要么使用传统的指标如ROUGE和BERTScore,这些指标依赖于表面相似性,无法考虑信息性,要么使用简单的LLM指标,这些指标不够稳健,容易被长上下文淹没。在本文中,我们提出了一种新的评估指标,称为EVA-Score,用于从给定的摘要中提取所有信息,根据参考识别重叠信息,并计算信息分数。我们测试了EVA-Score在多个数据集上的效果,实验结果表明,EVA-Score与人类的关联性最高。我们还从信息的角度重新评估了LLM在长形式摘要中的性能。结果表明,LLM的响应与人类撰写的答案之间仍存在差距。此外,我们还提供了EVA-Score有效性的详细分析,并预测了未来自动评估抽象长形式摘要的自动方法。

英文摘要:

Since LLMs emerged, more attention has been paid to abstractive long-form summarization, where longer input sequences indicate more information contained. Nevertheless, the automatic evaluation of such summaries remains underexplored. The current evaluation metrics for long-form summarization either use similarity-based metrics like ROUGE and BERTScore or LLM-based metrics using appropriate prompts or pre-defined schema. We argue that the former only relies on similarity and fails to consider informativeness while the latter lacks quantitative analysis of informative richness, and is rather subjective and hard to explain. Current evaluation metrics either use traditional metrics like ROUGE and BERTScore, which rely on surface-level similarity and fail to consider informativeness, or simple LLM-based metrics, which are not robust and easily overwhelmed by the long contexts. In this paper, we propose a new evaluation metric called EVA-Score to extract all information from the given summaries, identify overlapped information based on reference, and calculate the information score. We test EVA-Score on several datasets and the experimental results reveal that EVA-Score shows the highest correlation with humans. We also re-evaluate the performance of LLMs on long-form summarization from the information perspective. The results indicate that responses of LLMs still have a gap with the human-written answers. Moreover, we provide a detailed analysis of the effectiveness of EVA-Score, forecasting future ways to automatically evaluate abstractive long-form summarization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑