HistAlign:通过历史对齐改进语言生成中的上下文依赖性
HistAlign: Improving Context Dependency in Language Generation by Aligning with History
AI总结:
HistAlign 提出一种新的训练方法,通过改善缓存语言模型中当前隐藏状态与历史记忆的对齐,提升语言模型在提示续写、摘要和数据到文本等任务中的上下文依赖性与生成质量。
摘要:
语言模型(LMs)会产生幻觉和不连贯的输出,这凸显了它们较弱的上下文依赖性。缓存语言模型(Cache-LMs)通过为语言模型增加近期历史的记忆,可以增强上下文依赖性,并在多种语言生成任务中表现出显著性能。然而,我们发现,即使经过训练,当前缓存语言模型的缓存组件所带来的性能提升仍不理想,原因是当前隐藏状态与记忆中存储的隐藏状态之间存在错位。在这项工作中,我们提出了 HistAlign,一种新的训练方法,以确保良好的缓存对齐,从而使模型能够从历史中获得有用的信号。我们首先在一个简单且合成的任务上验证了我们的概念,在该任务中记忆对于正确预测至关重要,并且我们表明 HistAlign 的缓存组件对齐得更好,并提高了整体性能。接下来,我们在多种下游语言生成任务上评估 HistAlign,包括提示续写、抽象式摘要和数据到文本生成。我们证明,HistAlign 分别在开放式生成和条件式生成设置中提高了文本连贯性和忠实性。HistAlign 还能够在不同模型家族之间泛化,展示了其在多种场景下增强语言模型上下文依赖性的优势。我们的代码已在 https://github.com/meetdavidwan/histalign 公开。
英文摘要:
Language models (LMs) can generate hallucinations and incoherent outputs, which highlights their weak context dependency. Cache-LMs, which augment LMs with a memory of recent history, can increase context dependency and have shown remarkable performance in diverse language generation tasks. However, we find that even with training, the performance gain stemming from the cache component of current cache-LMs is suboptimal due to the misalignment between the current hidden states and those stored in the memory. In this work, we present HistAlign, a new training approach to ensure good cache alignment such that the model receives useful signals from the history. We first prove our concept on a simple and synthetic task where the memory is essential for correct predictions, and we show that the cache component of HistAlign is better aligned and improves overall performance. Next, we evaluate HistAlign on diverse downstream language generation tasks, including prompt continuation, abstractive summarization, and data-to-text. We demonstrate that HistAlign improves text coherence and faithfulness in open-ended and conditional generation settings respectively. HistAlign is also generalizable across different model families, showcasing its strength in improving context dependency of LMs in diverse scenarios. Our code is publicly available at https://github.com/meetdavidwan/histalign
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。