CNNSum：探索基于中文小说的大语言模型长上下文摘要
CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels
AI总结：
本文提出基于中文小说的长上下文摘要基准CNNSum，通过多模型评测与人工分析揭示现有LLMs在长摘要中的主观性、记忆依赖、提示敏感性及RoPE外推等问题，并验证短上下文数据微调可显著提升长摘要性能。
摘要：
大语言模型（LLMs）已在各类长上下文任务中得到了充分研究。然而，长上下文摘要数据集的稀缺阻碍了该领域的进展。为解决这一问题，我们提出了CNNSum，一个基于中文小说的多尺度长上下文摘要基准，包含四个子集、共695个样本，并带有人工标注，长度从16k到128k不等。我们对众多大语言模型进行了基准测试，并开展了详细的人工评估，以总结异常输出类型。此外，我们广泛探索了如何改进长上下文摘要。在我们的研究中：（1）先进的大语言模型可能生成大量主观评论，导致摘要含糊不清。（2）目前，长上下文摘要主要依赖记忆能力。大型LLMs的优势难以发挥，因此小型LLMs更具成本效益。（3）不同的提示类型与不同版本模型搭配可能造成较大的性能差距。在进一步微调中，这些差距可以得到缓解，且Base版本模型表现更好。（4）采用RoPE-base缩放的LLMs展现出较强的外推潜力；使用短上下文数据可以显著提升长上下文摘要性能。然而，进一步应用其他插值方法需要谨慎选择。（5）与其他基准相比，CNNSum提供了更可靠的评估结果。我们发布了CNNSum以推动未来研究。(https://github.com/CxsGhost/CNNSum)
英文摘要：
Large language models (LLMs) have been well-researched in various long-context tasks. However, the scarcity of long-context summarization datasets hinders progress in this area. To address this, we introduce CNNSum, a multi-scale long-context summarization benchmark based on Chinese novels, featuring human-driven annotations across four subsets totaling 695 samples, with lengths ranging from 16k to 128k. We benchmark numerous LLMs and conduct detailed human assessments to summarize abnormal output types. Furthermore, we extensively explore how to improve long-context summarization. In our study: (1) Advanced LLMs may generate much subjective commentary, leading to vague summaries. (2) Currently, long-context summarization mainly relies on memory ability. The advantages of Large LLMs are hard to utilize, thus small LLMs are more cost-effective. (3) Different prompt types paired with various version models may cause large performance gaps. In further fine-tuning, these can be mitigated, and the Base version models perform better. (4) LLMs with RoPE-base scaled exhibit strong extrapolation potential; using short-context data can significantly improve long-context summarization performance. However, further applying other interpolation methods requires careful selection. (5) CNNSum provides more reliable evaluation results than other benchmarks. We release CNNSum to advance future research.(https://github.com/CxsGhost/CNNSum)
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
