arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.00563cs.LGcs.AI
收藏

记忆决定学习方向:状态空间模型中基于梯度优化的理论

Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models

Intelligent Systems and Informatics Laboratory, Graduate School of Information Science and Technology, The University of Tokyo(智能系统与信息学实验室,信息科学与技术研究生院,东京大学)

JingChuan Guan, Tomoyuki Kubota, Yasuo Kuniyoshi, Kohei Nakajima

AI总结:

该研究针对状态空间模型(SSMs)学习动力学缺乏理论解释的问题,通过分析记忆容量特性阐明其学习机制,证明初始长记忆结构的重要性,发现固定循环权重可更快收敛且性能相当,为SSMs提供新理论基础与优化策略。

摘要:

状态空间模型(SSMs)因展现出超越Transformers的潜力而受到关注。然而,由于缺乏对SSMs学习动力学的理论解释,以往研究未能充分阐明其高性能的内在机制。本研究提供了这一解释,并提出了一种改进的训练策略。\nSSMs的记忆容量可通过检验输入时间序列如何存储在其当前状态中来评估。这一检验揭示了记忆准确性与长度之间的权衡,以及结构化状态空间序列模型(S4)与一种具有对角循环权重的简化S4之间的理论等价性。这一理论基础使我们得以阐明学习动力学,证明了初始参数的重要性。我们的分析结果表明,成功的学习需要初始记忆结构尽可能长,即使记忆准确性可能下降或梯度会丢失教师信息。\n在需要长记忆的任务上开展的实验证实,扩展记忆十分困难,这凸显了初始化的重要性。此外,我们发现固定循环权重可能比调整它们更具优势,因为前者能以更快的收敛速度达到相当甚至更高的性能。我们的研究结果为SSMs提供了新的理论基础,并有望提供一种新颖的优化策略。

英文摘要:

State space models (SSMs) have gained attention by showing potential to outperform Transformers. However, previous studies have not sufficiently addressed the mechanisms underlying their high performance owing to a lack of theoretical explanation of SSMs' learning dynamics. In this study, we provide such an explanation and propose an improved training strategy. The memory capacity of SSMs can be evaluated by examining how input time series are stored in their current state. Such an examination reveals a tradeoff between memory accuracy and length, as well as the theoretical equivalence between the structured state space sequence model (S4) and a simplified S4 with diagonal recurrent weights. This theoretical foundation allows us to elucidate the learning dynamics, proving the importance of initial parameters. Our analytical results suggest that successful learning requires the initial memory structure to be the longest possible even if memory accuracy may deteriorate or the gradient lose the teacher information. Experiments on tasks requiring long memory confirmed that extending memory is difficult, emphasizing the importance of initialization. Furthermore, we found that fixing recurrent weights can be more advantageous than adapting them because it achieves comparable or even higher performance with faster convergence. Our results provide a new theoretical foundation for SSMs and potentially offer a novel optimization strategy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑