通过正则化非局部泛函缓解Transformer中的过度平滑问题
Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals
AI总结：
本文提出NeuTRENO模型，通过在自注意力平滑输出与输入token之间引入正则化非局部泛函，有效缓解深度Transformer的过度平滑问题，并在分类、分割和语言建模任务中验证其优势。
摘要：
Transformer 在广泛的自然语言处理和计算机视觉应用中取得了显著成功。然而，深度 Transformer 模型的表示能力会因过度平滑问题而退化，即当模型深度增加时，token 表示会变得完全相同。在这项工作中，我们证明了 Transformer 中的自注意力层在最小化一个促进平滑性的泛函，从而导致 token 趋于一致。随后，我们提出了一种新颖的正则化器，该正则化器惩罚自注意力输出的平滑 token 与输入 token 之间差值的范数，以保持 token 的保真度。通过最小化由此得到的正则化能量泛函，我们推导出了带有正则化非局部泛函的神经 Transformer（NeuTRENO），这是一类能够缓解过度平滑问题的新型 Transformer 模型。我们通过实验证明了 NeuTRENO 相较于基线 Transformer 和当前最优方法在多种实际任务中的优势，这些任务包括物体分类、图像分割和语言建模，具体体现在减少 token 表示的过度平滑方面。
英文摘要：
Transformers have achieved remarkable success in a wide range of natural language processing and computer vision applications. However, the representation capacity of a deep transformer model is degraded due to the over-smoothing issue in which the token representations become identical when the model's depth grows. In this work, we show that self-attention layers in transformers minimize a functional which promotes smoothness, thereby causing token uniformity. We then propose a novel regularizer that penalizes the norm of the difference between the smooth output tokens from self-attention and the input tokens to preserve the fidelity of the tokens. Minimizing the resulting regularized energy functional, we derive the Neural Transformer with a Regularized Nonlocal Functional (NeuTRENO), a novel class of transformer models that can mitigate the over-smoothing issue. We empirically demonstrate the advantages of NeuTRENO over the baseline transformers and state-of-the-art methods in reducing the over-smoothing of token representations on various practical tasks, including object classification, image segmentation, and language modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
