arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00908cs.CLcs.AI
收藏

FineSurE:使用大语言模型进行细粒度摘要评估

FineSurE: Fine-grained Summarization Evaluation using LLMs

Korea Advanced Institute of Science and Technology(韩国科学技术院);AWS AI Labs(亚马逊云科技人工智能实验室)

Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, Saab Mansour

AI总结:

针对现有摘要评估方法粒度粗、维度单一的问题,提出基于大语言模型的细粒度评估器FineSurE,从忠实度、完整性和简洁性多维度评估,并在多个基准上超越现有方法。

摘要:

自动评估对于简化文本摘要基准测试和模型开发至关重要,因为人工评估成本高昂且耗时。传统方法如ROUGE与人类判断的相关性不佳,而最近提出的基于LLM的指标仅提供使用Likert量表分数的摘要级评估。这限制了更深入的模型分析,例如,我们只能在摘要级别分配一个幻觉分数,而在句子级别,我们可以统计包含幻觉的句子数量。为弥补这些限制,我们提出了FineSurE,一种专门针对摘要任务使用大语言模型(LLMs)的细粒度评估器。它除了忠实度外,还采用完整性和简洁性标准,实现多维度评估。我们比较了各种开源和专有LLM作为FineSurE的骨干模型。此外,我们针对包括基于NLI、QA和LLM的方法在内的SOTA方法进行了广泛的FineSurE基准测试,显示出在完整性和简洁性维度上的性能提升。代码可在https://github.com/DISL-Lab/FineSurE-ACL24获取。

英文摘要:

Automated evaluation is crucial for streamlining text summarization benchmarking and model development, given the costly and time-consuming nature of human evaluation. Traditional methods like ROUGE do not correlate well with human judgment, while recently proposed LLM-based metrics provide only summary-level assessment using Likert-scale scores. This limits deeper model analysis, e.g., we can only assign one hallucination score at the summary level, while at the sentence level, we can count sentences containing hallucinations. To remedy those limitations, we propose FineSurE, a fine-grained evaluator specifically tailored for the summarization task using large language models (LLMs). It also employs completeness and conciseness criteria, in addition to faithfulness, enabling multi-dimensional assessment. We compare various open-source and proprietary LLMs as backbones for FineSurE. In addition, we conduct extensive benchmarking of FineSurE against SOTA methods including NLI-, QA-, and LLM-based methods, showing improved performance especially on the completeness and conciseness dimensions. The code is available at https://github.com/DISL-Lab/FineSurE-ACL24.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑