arXiv 2604.256652026-06-23cs.CLcs.AIcs.DLcs.IR

LLM-ReSum: 一个通过自我评估实现LLM反思性总结的框架

LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation

dept. of Information Science University of North Texas Denton, Texas, USA（信息科学系俄克拉荷马州立大学丹顿俄克拉荷马州美国）；dept. of Data Science University of North Texas Denton, Texas, USA（数据科学系俄克拉荷马州立大学丹顿俄克拉荷马州美国）

Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Haihua Chen, Junhua Ding

备注：: This paper has been accepted as an invited paper for publication in Proceedings of The 12th IEEE International Conference on Big Data Computing Service and Machine Learning Applications. This is the accepted manuscript. The final authenticated version will be available via IEEE Xplore

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出LLM-ReSum框架，通过整合LLM评估与生成，提升低质量摘要的事实准确性与覆盖率，引入新的法律文档摘要基准PatentSumEval。

摘要：

可靠的评估大语言模型（LLM）生成的摘要仍是一个开放性挑战，特别是在异构领域和文档长度方面。我们对14种自动摘要指标和基于LLM的评估器进行了全面的元评估，覆盖七个数据集，涵盖从短新闻文章到长科学、政府和法律文本（2K-27K词）的文档，包含超过1500个人类标注的摘要。我们的结果表明，传统词汇重叠指标（如ROUGE、BLEU）与人类判断的关联性较弱或为负，而任务特定的神经指标和基于LLM的评估器在对语言质量的评估上表现更佳。基于这些发现，我们提出LLM-ReSum，一个自我反思的摘要框架，通过闭环反馈机制整合LLM评估与生成，无需模型微调。在三个领域中，LLM-ReSum在事实准确性上提升了高达33%，在覆盖范围上提升了39%，人类评估者在89%的情况下更倾向于改进后的摘要。我们还引入了PatentSumEval，一个包含180个专家评估摘要的新基准，用于法律文档摘要。所有代码和数据集将在GitHub上发布。

英文摘要：

Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。