恒定记忆召回：固定矩阵状态中的习得关联
Constant-Memory Recall: Learned Associations in a Fixed Matrix State
浏览论文内容
中文总结 AI 辅助
本研究探讨固定大小循环记忆的召回能力，提出带固定键偏置的DeltaNet变体，在32 KiB状态下以99.95%准确率召回32个键值对，但基线模型失败，无法比较记忆效率。
中文摘要 AI 辅助
固定大小的循环记忆限制了推理过程中的存储增长，但成功的召回取决于任务和训练。我们研究了一个带有固定令牌特定键偏置的小型DeltaNet变体，训练其记住每个序列中的32个新键值对。凭借32 KiB的循环矩阵状态，在三个训练种子中，当从序列的值中选择时，它达到了99.95%的平均准确率。当填充符将查询前上下文扩展到1,798个令牌而不添加配对时，召回仍接近完美。将第一个记忆块置零会消除这种召回。一项探索性的48对测试在主要训练预算的四分之一后仍接近随机水平，并未定位到容量极限。参数匹配的向量和Transformer基线仍接近随机水平，包括经过额外训练搜索后的Transformer。这一未解决的基线失败阻碍了记忆效率的比较。
英文摘要
Fixed-size recurrent memory limits storage growth during inference, but successful recall depends on the task and training. We study a small DeltaNet variant with fixed token-specific key biases, trained to remember 32 new key-value pairings per sequence. With 32 KiB of recurrent matrix state, it achieves 99.95% mean accuracy across three training seeds when choosing among the sequence's values. Recall remains near perfect when filler extends the pre-query context to 1,798 tokens without adding pairings. Zeroing the first memory block removes this recall. An exploratory 48-pair test remains near chance after one quarter of the primary training budget and does not locate a capacity limit. Parameter-matched vector and Transformer baselines remain near chance, including the Transformer after additional training searches. This unresolved baseline failure prevents a memory-efficiency comparison.
发表机构
- Pebble ML
机构由 AI 辅助整理，请以论文原文为准。
