arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.05573cs.LG
收藏

为什么深度在可并行化序列模型中重要:一个李代数视角

Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View

Howard Hughes Medical Institute, Department of Neurobiology, Harvard Medical School(霍华德·休斯医学研究所,哈佛医学院神经生物学系);Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University(自然与人工智能研究学院,哈佛大学);Department of Psychology and Center for Brain Science, Harvard University(心理学系和脑科学中心,哈佛大学)

Gyuryang Heo, Timothy Ngotiaoco, Kazuki Irie, Samuel J. Gershman, Bernardo L. Sabatini

AI总结:

从李代数控制视角,研究可并行化序列模型(如Transformer变体和状态空间模型)的表达能力与深度关系,证明误差随深度增加呈指数下降。

摘要:

可扩展的序列模型,如Transformer变体和结构化状态空间模型,通常以表达能力换取序列级并行性,从而实现高效训练。本文从李代数控制视角,考察模型在其表达能力范围之外运行时误差的边界及其缩放规律。我们的理论建立了序列模型深度与李代数扩展塔之间的对应关系。与近期理论研究相呼应,我们刻画了常数深度序列模型的李代数类别及其相应的表达能力边界。此外,我们解析推导了近似误差边界,并证明误差随深度增加呈指数下降,这与这些模型的强大实证表现一致。我们通过在符号词和连续值状态追踪问题上的实验验证了理论预测。

英文摘要:

Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑