深度随机变换器中的聚类
Clustering in Deep Stochastic Transformers
AI总结：
深度随机变换器中，初始化噪声导致标记在球面上形成相互作用粒子系统，而非单点聚类，且存在由相互作用强度和维度决定的相变现象。
摘要：
变换器已经彻底改变了各个领域的深度学习，但理解精确的标记动态仍然是一个理论挑战。现有的深度变换器层归一化理论通常预测标记会聚到一个点；然而，这些结果依赖于确定性权重假设，无法捕捉变换器中的标准初始化方案。在本工作中，我们展示了考虑随机初始化的内在随机性会改变这一图景。更具体地说，我们分析了深度变换器中噪声来自价值矩阵的随机初始化。在扩散缩放和标记级RMS归一化下，我们证明当变换器层数趋于无穷大时，离散标记动态收敛到一个在球面上的相互作用粒子系统，其中标记由一个共同的矩阵值布朗噪声驱动。在此极限下，我们证明初始化噪声防止了由确定性模型预测的单个聚类。对于两个标记，我们证明了一个由相互作用强度和标记维度决定的相变：不同于确定性注意力流，反向配置以正概率成为吸引子。数值实验证实了预测的相变，揭示了反向配置在超过两个标记时持续存在，并展示了抑制内在噪声会降低准确性。
英文摘要：
Transformers have revolutionized deep learning across various domains but understanding the precise token dynamics remains a theoretical challenge. Existing theories of deep Transformers with layer normalization typically predict that tokens cluster to a single point; however, these results rely on deterministic weight assumptions, which fail to capture the standard initialization scheme in Transformers. In this work, we show that accounting for the intrinsic stochasticity of random initialization alters this picture. More precisely, we analyze deep Transformers where noise arises from the random initialization of value matrices. Under diffusion scaling and token-wise RMS normalization, we prove that, as the number of Transformer layers goes to infinity, the discrete token dynamics converge to an interacting-particle system on the sphere where tokens are driven by a \emph{common} matrix-valued Brownian noise. In this limit, we show that initialization noise prevents the collapse to a single cluster predicted by deterministic models. For two tokens, we prove a phase transition governed by the interaction strength and the token dimension: unlike deterministic attention flows, antipodal configurations become attracting with positive probability. Numerical experiments confirm the predicted transition, reveal that antipodal formations persist for more than two tokens, and demonstrate that suppressing the intrinsic noise degrades accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
