arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.02823cs.LG
收藏

状态空间模型训练过程中压缩的奇特案例

The Curious Case of In-Training Compression of State Space Models

CSAIL, MIT(MIT 资深研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ETH Zürich(苏黎世联邦理工学院);ELLIS Institute(ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);Liquid AI

Makram Chahine, Philipp Nazari, Daniela Rus, T. Konstantin Rusch

AI总结:

CompreSSM通过在训练过程中压缩状态空间模型,提升计算效率同时保持性能。

摘要:

状态空间模型(SSMs)被开发用于高效地处理长序列建模任务,提供可并行化的训练和快速的推理。其核心是递归动力系统,维持隐藏状态,更新成本与状态维度成正比。关键的设计挑战是找到在最大化表达能力和限制此计算负担之间的合适平衡。控制理论,特别是Hankel奇异值分析,提供了一个强有力的框架来衡量每个状态的能量,以及将原始系统缩减到更小的表示,保证性能。利用Hankel矩阵的特征值稳定性性质,我们应用这一视角到SSMs的训练过程中,其中只有影响较大的维度被识别并保留。我们的方法,CompreSSM,适用于线性时不变的SSMs,如线性递归单元,但也可扩展到选择性模型。实验表明,训练过程中压缩显著加速了优化,同时保持了表达能力,压缩模型保留了任务关键的结构,而直接在较小维度上训练的模型则会丢失这些结构。换句话说,从大开始并随训练缩小的SSMs在计算效率方面取得平衡,同时保持了更高的性能。项目代码可在github.com/camail-official/compressm上获取。

英文摘要:

State Space Models (SSMs), developed to tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference. At their core are recurrent dynamical systems that maintain a hidden state, with update costs scaling with the state dimension. A key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden. Control theory, and more specifically Hankel singular value analysis, provides a potent framework for the measure of energy for each state, as well as the balanced truncation of the original system down to a smaller representation with performance guarantees. Leveraging the eigenvalue stability properties of Hankel matrices, we apply this lens to SSMs \emph{during training}, where only dimensions of high influence are identified and preserved. Our approach, \textsc{CompreSSM}, applies to Linear Time-Invariant SSMs such as Linear Recurrent Units, but is also extendable to selective models. Experiments show that in-training reduction significantly accelerates optimization while preserving expressivity, with compressed models retaining task-critical structure lost by models trained directly at smaller dimension. In other words, SSMs that begin large and shrink during training achieve computational efficiency while maintaining higher performance. Project code is available at github.com/camail-official/compressm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑