状态汤:上下文技能学习、检索与混合
State Soup: In-Context Skill Learning, Retrieval and Mixing
AI总结:
本文提出将循环模型内部状态视为可存储、检索和线性混合的任务向量,在 Mamba-2.8b 上验证简单状态插值即可提升困惑度与上下文学习性能。
摘要:
一类新型的门控线性循环神经网络已在一系列序列建模问题上达到最先进性能。此类模型天然能够高效处理长序列,因为处理新输入的成本与序列长度无关。在此,受参数插值实现模型合并这一成功经验的启发,我们探索这类有状态序列模型的另一优势。基于微调与上下文学习之间的相似性,我们研究是否可以将内部状态视为任务向量,从而利用循环的线性性质对其进行存储、检索和线性组合。我们在预训练循环模型 Mamba-2.8b 上研究这种快速模型合并形式,并给出初步证据表明,简单的线性状态插值方法足以改善下一 token 困惑度以及下游上下文学习任务性能。
英文摘要:
A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences efficiently, as the cost of processing a new input is independent of sequence length. Here, we explore another advantage of these stateful sequence models, inspired by the success of model merging through parameter interpolation. Building on parallels between fine-tuning and in-context learning, we investigate whether we can treat internal states as task vectors that can be stored, retrieved, and then linearly combined, exploiting the linearity of recurrence. We study this form of fast model merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary evidence that simple linear state interpolation methods suffice to improve next-token perplexity as well as downstream in-context learning task performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。