transformer模型中的布朗运动
The Brownian motion in the transformer model
摘要：
Transformer是许多语言和视觉任务的最新模型。本文对它的多头自注意（MHSA）模块进行了深入的分析，发现：1）每个标记在高维特征空间中是一个随机变量。2)在层归一化之后，这些变量被映射到超球体上的点。3)这些标记的更新是布朗运动。布朗运动具有特殊的性质，其二阶项是不可忽略的。因此，我们提出了一个新的二阶优化器（迭代K-FAC算法）的MHSA模块。简而言之：所有令牌都映射到高维超球体。标度点积注意力$softmax(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d}})$就是球上随机游动的马尔可夫转移矩阵。而深度学习过程将学习适当的核函数来获得这些标记的适当位置。MHSA模块中的训练过程对应于一个值得进一步研究的布朗运动。
英文摘要：
Transformer is the state of the art model for many language and visual tasks. In this paper, we give a deep analysis of its multi-head self-attention (MHSA) module and find that: 1) Each token is a random variable in high dimensional feature space. 2) After layer normalization, these variables are mapped to points on the hyper-sphere. 3) The update of these tokens is a Brownian motion. The Brownian motion has special properties, its second order item should not be ignored. So we present a new second-order optimizer(an iterative K-FAC algorithm) for the MHSA module. In some short words: All tokens are mapped to high dimension hyper-sphere. The Scaled Dot-Product Attention $softmax(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d}})$ is just the Markov transition matrix for the random walking on the sphere. And the deep learning process would learn proper kernel function to get proper positions of these tokens. The training process in the MHSA module corresponds to a Brownian motion worthy of further study.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
