Locas: 您的模型是局部支持的参数记忆的原理性初始化器
Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories
AI总结:
Locas通过原理化初始化方法,实现参数记忆的高效持续学习,减少灾难性遗忘,提升模型泛化能力。
摘要:
在本文中,我们旨在通过一种新的参数记忆类型来弥合测试时间训练与训练时间之间的差距,这种参数记忆可以灵活地从模型参数中卸载或合并到模型参数中。我们提出了Locas,一种局部支持的参数记忆,其设计与现代变压器中的FFN块相同,使其能够灵活地永久化到模型参数中,同时支持高效的持续学习。我们讨论了Locas的两个主要变种:一种采用传统的两层MLP设计,具有更清晰的理论保证;另一种与最先进的大语言模型(LLM)共享相同的GLU-FFN结构,可以轻松地附加到现有模型上,以实现参数高效和计算高效的持续学习。关键的是,我们展示了对这种低秩侧向FFN风格记忆的适当初始化——通过重用模型参数、激活值和/或梯度以原理化的方式进行初始化——对于快速收敛、改进泛化能力和防止灾难性遗忘是至关重要的。我们验证了所提出的记忆机制在PG-19全书语言建模和LoCoMo长上下文对话问答任务上的效果。在最低情况下,Locas-GLU仅需额外0.02%的参数,即可存储来自过去上下文的信息,同时保持更小的上下文窗口。此外,我们还通过比较MMLU评估测试模型在记忆全书后的一般能力损失。结果表明,Locas能够将过去的上下文永久化为参数知识,同时对模型现有内部知识的灾难性遗忘最小化。
英文摘要:
In this paper, we aim to bridge test-time-training with a new type of parametric memory that can be flexibly offloaded from or merged into model parameters. We present Locas, a Locally-Supported parametric memory that shares the design of FFN blocks in modern transformers, allowing it to be flexibly permanentized into the model parameters while supporting efficient continual learning. We discuss two major variants of Locas: one with a conventional two-layer MLP design that has a clearer theoretical guarantee; the other one shares the same GLU-FFN structure with SOTA LLMs, and can be easily attached to existing models for both parameter-efficient and computation-efficient continual learning. Crucially, we show that proper initialization of such low-rank sideway-FFN-style memories -- performed in a principled way by reusing model parameters, activations and/or gradients -- is essential for fast convergence, improved generalization, and catastrophic forgetting prevention. We validate the proposed memory mechanism on the PG-19 whole-book language modeling and LoCoMo long-context dialogue question answering tasks. With only 0.02\% additional parameters in the lowest case, Locas-GLU is capable of storing the information from past context while maintaining a much smaller context window. In addition, we also test the model's general capability loss after memorizing the whole book with Locas, through comparative MMLU evaluation. Results show the promising ability of Locas to permanentize past context into parametric knowledge with minimized catastrophic forgetting of the model's existing internal knowledge.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。