基于近似归一化Transformer的紧致空间学习
Learning in Compact Spaces with Approximately Normalized Transformer
AI总结：
针对深度网络训练需调优归一化超参数、现有超球面归一化有额外开销的问题，提出基于标量乘法的近似归一化与参数范数约束方法，在Transformer上实现更快收敛、低额外开销且保留GPT缩放特性。
摘要：
深度神经网络的成功训练需要解决过拟合、导致发散的数值不稳定性以及残差流方差增大等挑战。常见解决方案是采用正则化与归一化技术，这类方法通常需要调优额外超参数。另一种替代方案是强制所有参数和表征位于超球面上，该方式无需正则化且能提升收敛速度，但会带来额外开销。本研究受高维随机向量范数紧致集中特性启发，通过简单的标量乘法提出更具整体性的近似归一化方法；此外，我们不对参数施加严格归一化，而是约束其范数。这些改进无需权重衰减和学习率预热，且不会增加归一化层总数。我们在Transformer架构上开展的实验显示，与采用QK归一化的GPT模型相比，该方法收敛速度最高提升40%，仅带来3%的额外运行时开销。在推导缩放定律时我们发现，该方法支持使用更大批次大小进行训练，同时保留了经典GPT架构的优良缩放特性。
英文摘要：
The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
