Progress in summarizing long texts is inhibited by the lack of appropriate evaluation frameworks. When a long summary must be produced to appropriately cover the facets of that text, that summary needs to present a coherent narrative to be understandable by a reader, but current automatic and human evaluation methods fail to identify gaps in coherence. In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries. We develop a taxonomy of coherence errors in generated narrative summaries and collect span-level annotations for 6.6k sentences across 150 book and movie screenplay summaries. Our work provides the first characterization of coherence errors generated by state-of-the-art summarization models and a protocol for eliciting coherence judgments from crowd annotators. Furthermore, we show that the collected annotations allow us to train a strong classifier for automatically localizing coherence errors in generated summaries as well as benchmarking past work in coherence modeling. Finally, our SNaC framework can support future work in long document summarization and coherence evaluation, including improved summarization modeling and post-hoc summary correction.
Discourse Understanding and Factual Consistency in Abstractive Summarization
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Stanford University（斯坦福大学）；University of Cape Town（开普敦大学）
Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol
Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Department of Computer Science, Columbia University（哥伦比亚大学计算机科学系）；Department of Applied Physics and Applied Mathematics, Columbia University（哥伦比亚大学应用物理与应用数学系）；Department of Psychology, Northwestern University（西北大学心理学系）
Lay Text Summarisation Using Natural Language Processing: A Narrative Literature Review
Glasgow Caledonian University（格拉斯哥卡里多尼亚大学）；University of the West of Scotland（西苏格兰大学）；Lay Summaries Ltd（Lay Summaries 有限公司）；University of Edinburgh（爱丁堡大学）
以浅白的语言总结研究结果，对促进公众了解研究结果至关重要。使用自然语言处理生成非专业摘要有可能减轻研究人员的工作量，并弥合科学与社会之间的差距。本叙述性文献综述的目的是描述和比较用于生成非专业摘要的不同文本摘要方法。我们检索了数据库Web of Science、Google Scholar、IEEE Xplore、Association for Computing Machinery数字图书馆和arXiv，查找截至2022年5月6日发表的文章。我们纳入了关于自动文本摘要方法的原始研究，以生成非专业摘要。我们筛选了82篇文章，并纳入了2020年至2021年期间发表的8篇相关论文，全部使用相同的数据集。结果表明，基于transformer的方法，如双向编码器表示从Transformers（BERT）和预训练提取的差距句子抽象摘要（PEGASUS）占主导地位的景观奠定文本摘要，所有的研究，但一个使用这些方法。在混合方法中，提取和抽象总结方法的组合被认为是最有效的。此外，输入文本的预处理方法（例如，应用提取摘要）或确定文本的哪些部分将被包括在内，显得至关重要。使用了不考虑可读性的评价指标，如面向回忆的替代评价（ROUGE）。总而言之，自动的文本摘要还没有得到充分的开发。未来的研究应考虑长文档的非专业文本摘要，包括临床试验报告，以及考虑非专业摘要可读性的评估指标的开发。
Dual-stage and Lightweight Patient Chart Summarization for Emergency Physicians
Department of Electrical and Software Engineering, University of Calgary, Calgary, AB, Canada（电气与软件工程系，卡尔加里大学）；Department of Emergency Medicine, University of Calgary, Calgary, AB, Canada（急诊医学系，卡尔加里大学）；School of Information, University of Michigan, Ann Arbor, MI, USA（信息学院，密歇根大学）；Rockview General Hospital, Calgary, AB, Canada（罗克维尔总医院）
