LPC-SM:局部预测编码与稀疏记忆用于长上下文语言建模
LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling
AI总结:
本文提出LPC-SM架构,通过分离局部注意力、持久记忆、预测修正和运行时控制,改进长上下文语言模型的性能,实验显示其在不同任务中均取得显著提升。
摘要:
当前长上下文语言模型仍依赖注意力处理局部交互和长程状态,限制了测试其他序列建模分解方法的机会。我们提出LPC-SM,一种混合自回归架构,将局部注意力、持久记忆、预测修正和运行时控制整合在同一模块中,并使用正交新颖性传输(ONT)管理慢速记忆写入。我们评估了一个15800万参数模型,分三个阶段测试基础语言建模、数学延续和4096个token延续。移除mHC使Stage-A最终LM损失从12.630升至15.127,而自适应稀疏控制将Stage-B最终LM损失从12.137降至10.787,相对于匹配的固定比率延续。完整路线在序列长度4096处保持稳定,Stage C以最终LM损失11.582结束,并将关键交叉熵的延迟标识诊断从14.396降至12.031。这些结果表明,长上下文自回归建模可以围绕比注意力本身更广泛的分工组织。
英文摘要:
Most current long-context language models still rely on attention to handle both local interaction and long-range state, which leaves relatively little room to test alternative decompositions of sequence modeling. We propose LPC-SM, a hybrid autoregressive architecture that separates local attention, persistent memory, predictive correction, and run-time control within the same block, and we use Orthogonal Novelty Transport (ONT) to govern slow-memory writes. We evaluate a 158M-parameter model in three stages spanning base language modeling, mathematical continuation, and 4096-token continuation. Removing mHC raises the Stage-A final LM loss from 12.630 to 15.127, while adaptive sparse control improves the Stage-B final LM loss from 12.137 to 10.787 relative to a matched fixed-ratio continuation. The full route remains stable at sequence length 4096, where Stage C ends with final LM loss 11.582 and improves the delayed-identifier diagnostic from 14.396 to 12.031 in key cross-entropy. Taken together, these results show that long-context autoregressive modeling can be organized around a broader division of labor than attention alone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。