词汇重复导致死记硬背：揭示训练与测试参考摘要中词汇重叠的影响
Lexical Repetitions Lead to Rote Learning: Unveiling the Impact of Lexical Overlap in Train and Test Reference Summaries
AI总结：
本文提出细粒度评估协议，发现训练与测试摘要词汇重叠导致模型死记硬背，限制训练摘要词汇重复可提升泛化能力。
摘要：
理想的摘要模型应能泛化到新的值得摘要的内容，而无需死记硬背训练参考摘要。然而，整个测试集上的单一平均性能分数不足以确定这种模型能力。我们提出了一种细粒度的评估协议，通过将测试集基于参考测试摘要与训练摘要的词汇相似性进行划分。我们观察到，在相似性最低和最高的子集之间，ROUGE-2（实体召回率）分数存在高达5倍（1.2倍）的差异。接下来，我们表明这种训练重复也使模型容易受到死记硬背的影响，重现数据伪影，如事实错误，特别是当参考测试摘要与训练摘要在词汇上接近时。因此，我们建议在监督微调和似然校准阶段限制训练摘要中的词汇重复，以提高在新测试案例上的性能，同时保持平均性能。我们在新测试子集和近期新闻文章上的自动和人工评估表明，限制训练摘要中的词汇重复可以防止死记硬背并提高泛化能力。
英文摘要：
Ideal summarization models should generalize to novel summary-worthy content without remembering reference training summaries by rote. However, a single average performance score on the entire test set is inadequate in determining such model competencies. We propose a fine-grained evaluation protocol by partitioning a test set based on the lexical similarity of reference test summaries with training summaries. We observe up to a 5x (1.2x) difference in ROUGE-2 (entity recall) scores between the subsets with the lowest and highest similarity. Next, we show that such training repetitions also make a model vulnerable to rote learning, reproducing data artifacts such as factual errors, especially when reference test summaries are lexically close to training summaries. Consequently, we propose to limit lexical repetitions in training summaries during both supervised fine-tuning and likelihood calibration stages to improve the performance on novel test cases while retaining average performance. Our automatic and human evaluations on novel test subsets and recent news articles show that limiting lexical repetitions in training summaries can prevent rote learning and improve generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
