激活记忆与参数化记忆在少样本学习中的互补作用
Complementary Roles of Activation and Parametric Memory in Few-Shot Learning
- Northeastern University（东北大学）
- NiuTrans Research（NiuTrans 研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究通过受控实验发现，大型语言模型在少样本学习中，激活记忆擅长事实回忆，参数化记忆并非始终更优，且复合任务需两者协同，神经元分析证实其互补性。
AI中文摘要：
在测试时，大型语言模型（LLMs）可以将历史信息编码在激活记忆（即KV缓存）和参数化记忆（即更新的参数）中。虽然激活记忆通常被认为对事实回忆有效，而参数化记忆对学习新任务有效，但两者之间的相互作用仍不清楚。在本工作中，我们通过受控实验系统地研究了少样本学习中记忆的作用。我们发现，激活记忆在回忆事实方面更优，而参数化记忆在任务学习方面并不始终优于激活记忆。此外，我们的实验表明，复合任务——条件算术——需要两种记忆类型的协同作用。通过神经元层面的分析，我们发现，当通过激活记忆与参数化记忆访问相同的历史信息时，模型激活了不同的神经元集合。当两种记忆类型结合时，模型从两个集合中招募神经元，这对解决条件算术至关重要。这些发现表明，单独任何一种记忆机制都不足以完成这一复合任务，凸显了它们协作的重要性。
英文摘要：
At test time, large language models (LLMs) can encode historical information in activation memory (i.e., KV caches) and parametric memory (i.e., updated parameters). While activation memory is generally considered effective for factual recall and parametric memory for learning new tasks, their interplay remains unclear. In this work, we systematically investigate the role of memory in few-shot learning through controlled experiments. We find that activation memory is superior for recalling facts, whereas parametric memory does not consistently outperform activation memory in task learning. Moreover, our experiments show that the composite task, Conditional Arithmetic, requires the synergy of both memory types. Through neuron-level analysis, we find that the model activates distinct sets of neurons when accessing the same historical information through activation versus parametric memory. When both memory types are combined, the model recruits neurons from both sets, which is crucial for solving Conditional Arithmetic. These findings suggest that neither memory mechanism alone is sufficient for this composite task, highlighting the importance of their collaboration.
