梯度定位改善语言模型的终身预训练
Gradient Localization Improves Lifelong Pretraining of Language Models
AI总结：
本研究通过分析时间敏感实体知识在LLM中的参数局部性，提出将参数更新定向到相关层，以改善持续预训练中时间漂移语言的性能。
摘要：
在网页规模文本语料上训练的大型语言模型（LLMs）已被证明能够在其参数中捕获世界知识。然而，语言模型存储不同类型知识的机制尚不明确。在本工作中，我们考察了与时间敏感实体相关的两类知识，并证明每种知识都定位在LLMs内不同的参数集中。我们假设，现有持续学习方法缺乏对知识局部性的考虑，这既导致了新信息吸收的失败，也导致了先前学习信息的灾难性遗忘。我们观察到，包含对更新和新提及实体引用的序列在部分层中表现出更大的梯度范数。我们证明，将参数更新针对这些相关层，可以改善对含有时间漂移语言的持续预训练性能。
英文摘要：
Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store different types of knowledge is poorly understood. In this work, we examine two types of knowledge relating to temporally sensitive entities and demonstrate that each type is localized to different sets of parameters within the LLMs. We hypothesize that the lack of consideration of the locality of knowledge in existing continual learning methods contributes to both: the failed uptake of new information, and catastrophic forgetting of previously learned information. We observe that sequences containing references to updated and newly mentioned entities exhibit larger gradient norms in a subset of layers. We demonstrate that targeting parameter updates to these relevant layers can improve the performance of continually pretraining on language containing temporal drift.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
