arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01698cs.LGstat.ML
收藏

通过近似对角化增强状态空间模型对长序列的鲁棒性

Robustifying State-space Models for Long Sequences via Approximate Diagonalization

Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);International Computer Science Institute(国际计算机科学研究所);Cornell University(康奈尔大学);University of California at Berkeley(加州大学伯克利分校)

Annan Yu, Arnur Nigmetov, Dmitriy Morozov, Michael W. Mahoney, N. Benjamin Erichson

AI总结:

针对HiPPO框架对角化的不适定问题,提出基于非正规算子伪谱理论的通用PTD方法,构建S4-PTD/S5-PTD模型,兼具更强的噪声鲁棒性与更优的长序列任务准确率。

摘要:

状态空间模型(SSMs)近期已成为学习长程序列任务的一类框架,结构化状态空间序列(S4)层便是其中一例,它采用了HiPPO初始化框架的对角加低秩结构。但S4层的复杂结构带来了挑战;为应对这些挑战,S4D、S5等模型采用了纯对角结构。这一选择简化了实现、提升了计算效率,还支持通道通信。然而,对HiPPO框架进行对角化本身是一个不适定问题。\n本文针对该问题及机器学习中相关的不适定对角化问题提出了通用解决方案。我们提出了一种通用的、后向稳定的“扰动后对角化”(perturb-then-diagonalize, PTD)方法,该方法基于非正规算子的伪谱理论,可理解为对定义SSM的非正规矩阵进行近似对角化。基于此,我们提出了S4-PTD和S5-PTD模型。\n通过对不同初始化方案的传递函数进行理论分析,我们证明S4-PTD/S5-PTD初始化能强收敛到HiPPO框架,而S4D/S5初始化仅能实现弱收敛。因此,我们的新模型对傅里叶模式噪声扰动的输入具有恢复能力,这是S4D/S5模型不具备的关键特性。除鲁棒性提升外,我们的S5-PTD模型在Long-Range Arena基准上的平均准确率达到87.6%,证明PTD方法有助于提升深度学习模型的准确率。

英文摘要:

State-space models (SSMs) have recently emerged as a framework for learning long-range sequence tasks. An example is the structured state-space sequence (S4) layer, which uses the diagonal-plus-low-rank structure of the HiPPO initialization framework. However, the complicated structure of the S4 layer poses challenges; and, in an effort to address these challenges, models such as S4D and S5 have considered a purely diagonal structure. This choice simplifies the implementation, improves computational efficiency, and allows channel communication. However, diagonalizing the HiPPO framework is itself an ill-posed problem. In this paper, we propose a general solution for this and related ill-posed diagonalization problems in machine learning. We introduce a generic, backward-stable "perturb-then-diagonalize" (PTD) methodology, which is based on the pseudospectral theory of non-normal operators, and which may be interpreted as the approximate diagonalization of the non-normal matrices defining SSMs. Based on this, we introduce the S4-PTD and S5-PTD models. Through theoretical analysis of the transfer functions of different initialization schemes, we demonstrate that the S4-PTD/S5-PTD initialization strongly converges to the HiPPO framework, while the S4D/S5 initialization only achieves weak convergences. As a result, our new models show resilience to Fourier-mode noise-perturbed inputs, a crucial property not achieved by the S4D/S5 models. In addition to improved robustness, our S5-PTD model averages 87.6% accuracy on the Long-Range Arena benchmark, demonstrating that the PTD methodology helps to improve the accuracy of deep learning models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑