为什么深度在可并行化序列模型中重要:一个李代数视角
Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic View
AI总结:
从李代数控制视角,研究可并行化序列模型(如Transformer变体和状态空间模型)的表达能力与深度关系,证明误差随深度增加呈指数下降。
摘要:
可扩展的序列模型,如Transformer变体和结构化状态空间模型,通常以表达能力换取序列级并行性,从而实现高效训练。本文从李代数控制视角,考察模型在其表达能力范围之外运行时误差的边界及其缩放规律。我们的理论建立了序列模型深度与李代数扩展塔之间的对应关系。与近期理论研究相呼应,我们刻画了常数深度序列模型的李代数类别及其相应的表达能力边界。此外,我们解析推导了近似误差边界,并证明误差随深度增加呈指数下降,这与这些模型的强大实证表现一致。我们通过在符号词和连续值状态追踪问题上的实验验证了理论预测。
英文摘要:
Scalable sequence models, such as Transformer variants and structured state-space models, often trade expressivity power for sequence-level parallelism, which enables efficient training. Here we examine the bounds on error and how error scales when models operate outside of their expressivity regimes using a Lie-algebraic control perspective. Our theory formulates a correspondence between the depth of a sequence model and the tower of Lie algebra extensions. Echoing recent theoretical studies, we characterize the Lie-algebraic class of constant-depth sequence models and their corresponding expressivity bounds. Furthermore, we analytically derive an approximation error bound and show that error diminishes exponentially as the depth increases, consistent with the strong empirical performance of these models. We validate our theoretical predictions using experiments on symbolic word and continuous-valued state-tracking problems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。