arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.22329cs.LGcs.AI
收藏

训练的持久内存用于冻结的解码器-only LLMs

Trained Persistent Memory for Frozen Decoder-Only LLMs

Inha University in Tashkent, Uzbekistan(乌兹别克斯坦塔什克恩的因哈大学)

Hong Jeong

AI总结:

本文研究解码器-only LLMs中持久内存的实现,通过六种方法在冻结的GPT-2上训练小型适配器,发现架构先验对保留内存性能有显著影响,且在10倍容量下所有方法均收敛。

摘要:

解码器-only语言模型是无状态的:隐藏表示在每次前向传递后都被丢弃,无法在会话间持续。Jeong (2026a)表明,训练的内存适配器可以在冻结的编码器-解码器基础架构上提供持久的潜在空间记忆,基于Jeong (2026b,c)的横向记忆框架。本文探讨是否在解码器-only设置中同样适用,其中没有交叉注意力路径,记忆必须通过自注意力单独进入。我们将六种方法——前缀、并行交叉注意力、KV扩展、Hebbian记忆、上下文门控分支和基于槽位的稀疏写入——应用于冻结的GPT-2,仅训练小型适配器θ_mem。写规则相同;只有读取注入从解码器交叉注意力变为自注意力KV前缀或并行分支。在LoCoMo上发现显著的归纳偏差二元性:在1×容量下,三种具有强架构先验的方法——交叉注意力(M.2)、Hebbian(M.4)和槽位写入(M.6)——实现了保留内存分数为7-18%和知识增益ΔK为7-10,而其他三种方法失败(<0.4%)。在10×容量下所有六种方法均收敛,表明差距是架构性的,而非根本性的。结合Jeong (2026a)的编码器-解码器结果和Jeong (2026b,c)的脑启发模块,这些发现确立了持久潜在空间记忆作为跨越主要Transformer家族的通用范式。

英文摘要:

Decoder-only language models are stateless: hidden representations are discarded after every forward pass and nothing persists across sessions. Jeong (2026a) showed that trained memory adapters give a frozen encoder-decoder backbone persistent latent-space memory, building on the lateral-memory framework of Jeong (2026b,c). Here we ask whether the same principle transfers to the decoder-only setting, where no cross-attention pathway exists and memory must enter through self-attention alone. We adapt six methods -- prefix, parallel cross-attention, KV extension, Hebbian memory, context-gated branch, and slot-based sparse write -- to a frozen GPT-2, training only a small adapter $θ_{mem}$. The write rule is shared; only the read injection changes from decoder cross-attention to self-attention KV prefix or parallel branch. On LoCoMo we find a striking inductive-bias dichotomy: at $1\times$ capacity, three methods with strong architectural priors -- cross-attention (M.2), Hebbian (M.4), and slot write (M.6) -- achieve retained-memory scores of $7-18\%$ and knowledge gains $ΔK$ of $7-10$, while the other three fail ($< 0.4\%$). At $10\times$ capacity all six converge, showing the gap is architectural, not fundamental. Together with the encoder-decoder results of Jeong (2026a) and the brain-inspired modules of Jeong (2026b,c), these findings establish persistent latent-space memory as a general paradigm spanning major transformer families.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑