arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.01701cs.CL
收藏

论摘要内容单元在文本摘要评估中的作用

On the Role of Summary Content Units in Text Summarization Evaluation

TU Darmstadt(达姆施塔特工业大学);University of Zurich(苏黎世大学);Amazon AGI(亚马逊AGI);New York University(纽约大学);Google Research(谷歌研究院);ADAPT Centre, DCU(都柏林城市大学ADAPT中心);Yale University(耶鲁大学);Bloomberg(彭博公司);trivago N.V.(trivago公司);University of Edinburgh(爱丁堡大学);Allen Institute for AI(艾伦人工智能研究所);IBM Research Europe, Ireland(IBM欧洲研究院爱尔兰分部)
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining… 展开作者

Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C. Walenta, Juri Opitz, Leonardo F. R. Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou

AI总结:

本文研究 SCU 及其自动近似在 Pyramid 摘要评估中的作用,提出基于 AMR 和 LLM 的 SMU、SGU,并发现 SGU 近似质量最佳且 SCU 更适用于短摘要排序。

摘要:

文本摘要的 Pyramid 评估方法的核心是人工撰写的摘要内容单元(SCU)。这些 SCU 是将摘要分解为若干小事实的简洁句子,可用于判断候选摘要的质量,并可能通过自然语言推理(NLI)系统实现部分自动化。有趣的是,为了实现 Pyramid 评估的完全自动化,Zhang 和 Bansal(2021)表明,SCU 可以用自动生成的语义角色三元组(STU)来近似。然而,目前仍有若干问题没有答案,尤其是:i)是否存在其他近似 SCU 的方式并能带来优势?ii)SCU(或其近似形式)在什么条件下能提供最大价值?在本研究中,我们考察了两种用于近似 SCU 的新策略:分别从 AMR 意义表示生成 SCU 近似形式(SMU),以及从大型语言模型生成 SCU 近似形式(SGU)。我们发现,尽管 STU 和 SMU 具有竞争力,但最佳近似质量由 SGU 实现。我们还通过一个简单的句子分解基线(SSU)表明,SCU(及其近似形式)在对短摘要进行排序时价值最大,但在对系统或较长摘要进行排序时帮助可能没有那么大。

英文摘要:

At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑