一个显式算子解释了现代用于序列和语言建模的神经网络的端到端计算
An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
AI总结:
本文通过建立状态空间模型与可解非线性振荡网络的数学对应关系,揭示了S4D模型的端到端计算机制,并展示了其在序列分类中的作用。
摘要:
我们建立了状态空间模型与可解非线性振荡网络之间的数学对应关系。作为具体例子,我们分析了结构化状态空间序列模型(S4)的对角线线性时不变实现。该对应关系将S4D嵌入到环形网络拓扑中,近期输入被编码为在网络一维空间布局上传播的活动波。我们推导了S4D完整前向传递的精确算子表达式,从而获得了其完整输入输出映射的分析描述。该表达式揭示了系统中的非线性解码器诱导了这些信息承载波之间的相互作用,从而实现了对现实序列的分类。这些结果适用于现代SSM架构,表明它们具有精确的数学描述和清晰的物理解释。这些见解为这些系统在非线性振荡网络方面的可解释性提供了新的层面。
英文摘要:
We establish a mathematical correspondence between state space models, a state-of-the-art architecture for capturing long-range dependencies in data, and an exactly solvable nonlinear oscillator network. As a specific example of this general correspondence, we analyze the diagonal linear time-invariant implementation of the Structured State Space Sequence model (S4). The correspondence embeds S4D, a specific implementation of S4, into a ring network topology, in which recent inputs are encoded, as waves of activity traveling over the one-dimensional spatial layout of the network. We then derive an exact operator expression for the full forward pass of S4D, yielding an analytical characterization of its complete input-output map. This expression reveals that the nonlinear decoder in the system induces interactions between these information-carrying waves that enable classifying real-world sequences. These results generalize across modern SSM architectures, and show that they admit an exact mathematical description with a clear physical interpretation. These insights enable a new level of interpretability for these systems in terms of nonlinear oscillator networks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。