逃离梯度消失：Softmax在注意力机制中的周期性选择
Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism
摘要：
Softmax广泛用于神经网络的多类分类、门结构和注意力机制。输入是正态分布的统计假设支持Softmax的梯度稳定性。但是，当用于Transformers等注意力机制时，由于嵌入之间的相关性得分往往不是正态分布的，因此出现了梯度消失问题，我们通过实验确认证明了这一点。在这项工作中，我们建议用周期函数代替指数函数，并从值和梯度的角度深入研究了Softmax的一些潜在的周期替代方案。通过在参考LeViT的简单设计的演示上的实验，我们的方法被证明能够缓解梯度问题，并与Softmax及其变体相比产生实质性的改进。此外，我们通过数学和实验分析了预归一化对Softmax和我们的方法的影响。最后，我们增加了演示的深度，并证明了我们的方法在深部结构的适用性。
英文摘要：
Softmax is widely used in neural networks for multiclass classification, gate structure and attention mechanisms. The statistical assumption that the input is normal distributed supports the gradient stability of Softmax. However, when used in attention mechanisms such as transformers, since the correlation scores between embeddings are often not normally distributed, the gradient vanishing problem appears, and we prove this point through experimental confirmation. In this work, we suggest that replacing the exponential function by periodic functions, and we delve into some potential periodic alternatives of Softmax from the view of value and gradient. Through experiments on a simply designed demo referenced to LeViT, our method is proved to be able to alleviate the gradient problem and yield substantial improvements compared to Softmax and its variants. Further, we analyze the impact of pre-normalization for Softmax and our methods through mathematics and experiments. Lastly, we increase the depth of the demo and prove the applicability of our method in deep structures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
