连续记忆机器
Continuous Memory Machines
浏览论文内容
中文总结 AI 辅助
针对循环神经网络记忆瓶颈，提出连续记忆机器（CMM），采用矩阵值短期与长期记忆分离，通过Transformer双向读写，在算法、上下文学习及推理任务上超越基线，兼具强泛化与可解释注意力。
中文摘要 AI 辅助
循环神经网络通常将信息压缩为单一向量值的循环状态，迫使短期计算和长期保留共享同一表示。过去的扩展通过增加记忆容量或分离时间尺度来缓解这一瓶颈，但缺乏生物学中发现的快速神经元级处理与长期保留的结合。为此，我们引入了连续记忆机器（CMM），一种具有矩阵值短期和长期记忆状态、分别承担不同功能角色的循环架构。基于连续思维机器（CTM），CMM的短期记忆追踪近期神经活动，通过独特参数化的神经元级模型学习利用这些活动模式进行计算。持久长期记忆存储信息以供后续使用，Transformer联合更新两个记忆存储，提供表达性强的双向读写机制，使得每个存储可以重组自身内容，并同时从另一个存储读取和写入。在算法任务、上下文学习任务和循环推理任务中，CMM优于广泛的基线模型，展现出比先前记忆增强网络更强的泛化能力，同时保留了CTM的可解释注意力模式。代码可在https URL获取。
英文摘要
Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read--write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns. Code is available at https://github.com/SakanaAI/continuous-memory-machines.
发表机构
- University of Tsukuba（筑波大学）
- Sakana AI
机构由 AI 辅助整理，请以论文原文为准。
