LipsFormer：将 Lipschitz 连续性引入 Vision Transformers
LipsFormer: Introducing Lipschitz Continuity to Vision Transformers
AI总结：
本文提出 Lipschitz 连续的 LipsFormer，通过替换归一化、初始化、注意力和残差连接实现无预热的稳定深度 Transformer 训练，并在 ImageNet-1K 上取得更高精度。
摘要：
我们提出了一种 Lipschitz 连续的 Transformer，称为 LipsFormer，旨在从理论和经验两方面为基于 Transformer 的模型追求训练稳定性。以往的实用技巧通过学习率预热、层归一化、注意力公式和权重初始化来解决训练不稳定性；与之不同，我们表明 Lipschitz 连续性是确保训练稳定性的一项更本质的属性。在 LipsFormer 中，我们用 Lipschitz 连续的对应模块替换不稳定的 Transformer 组件：用 CenterNorm 替代 LayerNorm，用谱初始化替代 Xavier 初始化，用缩放余弦相似度注意力替代点积注意力，并采用加权残差捷径。我们证明了这些引入的模块是 Lipschitz 连续的，并推导了 LipsFormer 的 Lipschitz 常数上界。实验表明，LipsFormer 能够稳定训练深度 Transformer 架构，而无需预热等精细的学习率调节，从而带来更快的收敛和更好的泛化。因此，在 ImageNet 1K 数据集上，基于 Swin Transformer 的 LipsFormer-Swin-Tiny 训练 300 个 epoch，在没有任何学习率预热的情况下可达到 82.7%。此外，基于 CSwin 的 LipsFormer-CSwin-Tiny 训练 300 个 epoch，以 4.7G FLOPs 和 24M 参数取得了 83.5% 的 top-1 准确率。代码将发布于 https://github.com/IDEA-Research/LipsFormer。
英文摘要：
We present a Lipschitz continuous Transformer, called LipsFormer, to pursue training stability both theoretically and empirically for Transformer-based models. In contrast to previous practical tricks that address training instability by learning rate warmup, layer normalization, attention formulation, and weight initialization, we show that Lipschitz continuity is a more essential property to ensure training stability. In LipsFormer, we replace unstable Transformer component modules with Lipschitz continuous counterparts: CenterNorm instead of LayerNorm, spectral initialization instead of Xavier initialization, scaled cosine similarity attention instead of dot-product attention, and weighted residual shortcut. We prove that these introduced modules are Lipschitz continuous and derive an upper bound on the Lipschitz constant of LipsFormer. Our experiments show that LipsFormer allows stable training of deep Transformer architectures without the need of careful learning rate tuning such as warmup, yielding a faster convergence and better generalization. As a result, on the ImageNet 1K dataset, LipsFormer-Swin-Tiny based on Swin Transformer training for 300 epochs can obtain 82.7\% without any learning rate warmup. Moreover, LipsFormer-CSwin-Tiny, based on CSwin, training for 300 epochs achieves a top-1 accuracy of 83.5\% with 4.7G FLOPs and 24M parameters. The code will be released at \url{https://github.com/IDEA-Research/LipsFormer}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
