发表机构
University of Wisconsin–Madison; University of Maryland; University of Warsaw; Rice University; University of Oxford（威斯康星大学麦迪逊分校; 马里兰大学; 华沙大学; 莱斯大学; 牛津大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究深度线性编码器-only变压器推理时行为，通过相互作用粒子系统视角，在二维嵌入维度揭示其动力学可表述为广义Kuramoto型模型，还发现多种长期行为及相关特性，高维数值实验表明这些行为在更高维度持续存在。
AI 中文摘要
我们通过相互作用粒子系统的视角研究深度线性编码器-only变压器的推理时行为。在此视角下，令牌被建模为通过连续线性自注意力层动态相互作用的粒子。我们表明，在二维嵌入维度中，对于任何键、查询和值矩阵，动力学可重新表述为具有纯二次谐波耦合的广义Kuramoto型模型。这种表述适用于渡边-斯特罗加茨理论，该理论揭示动力学本质上是低维的，与参数矩阵无关。对于与奥-安滕森（OA）流形相关的一类令牌初始化，我们表明参数矩阵在线性变压器中诱导出多种长期行为，包括聚类、振荡和分岔。振荡和分岔通过揭示动力学中的隐藏哈密顿结构来表征。通过建立结构稳定性结果，我们进一步表明在OA流形附近初始化的动力学表现出与在流形上精确初始化的动力学相同的长期行为。受二维理论的启发，我们对高维变压器中的类似参数区域进行了数值实验。我们的数值实验表明，我们理论结果中表征的长期行为在更高维度中持续存在。
英文摘要
We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter matrices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that the parameter matrices induce a diverse variety of long-time behaviors in linear transformers, including clustering, oscillations, and bifurcations. The oscillations and bifurcations are characterized by uncovering a hidden Hamiltonian structure in the dynamics. By establishing a structural stability result, we further show that dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold. Motivated by our theory in dimension two, we conduct numerical experiments for analogous parameter regimes in higher-dimensional transformers. Our numerical experiments suggest that the long-time behaviors characterized in our theoretical results persist in higher dimensions.
