SimpleGPT：通过一种简单的归一化策略改进GPT
SimpleGPT: Improving GPT via A Simple Normalization Strategy
AI总结：
SimpleGPT通过SimpleNorm策略提升GPT模型的训练稳定性与性能
摘要：
在本文中，我们通过二阶几何的角度重新审视Transformer的优化，并建立了架构设计、激活尺度、Hessian矩阵和最大可容忍学习率之间的直接联系。我们引入了一种简单的归一化策略，称为SimpleNorm，该策略通过构造方式稳定中间激活尺度。然后，通过分析损失对网络激活的Hessian矩阵，我们理论证明SimpleNorm显著降低了Hessian的谱范数，从而允许更大的稳定学习率。我们通过在1B、1.4B、7B和8B参数规模的大型GPT模型上的广泛实验验证了我们的理论发现。实证上，我们的SimpleNorm基于网络SimpleGPT能够容忍比标准惯例大3至10倍的学习率，一致表现出强大的优化稳定性，并在已建立的基准上取得了显著更好的性能。具体来说，当训练7B规模模型6万步时，SimpleGPT的训练损失比使用QKNorm的LLaMA2低0.08，将损失从2.290降低到2.208。我们的源代码将在https://github.com/Ocram7/SimpleGPT上发布。
英文摘要：
In this work, we revisit Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the Hessian matrix, and the maximum tolerable learning rate. We introduce a simple normalization strategy, termed SimpleNorm, which stabilizes intermediate activation scales by construction. Then, by analyzing the Hessian of the loss with respect to network activations, we theoretically show that SimpleNorm significantly reduces the spectral norm of the Hessian, thereby permitting larger stable learning rates. We validate our theoretical findings through extensive experiments on large GPT models at parameter scales 1B, 1.4B, 7B and 8B. Empirically, SimpleGPT, our SimpleNorm-based network, tolerates learning rates 3$\times$-10$\times$ larger than standard convention, consistently demonstrates strong optimization stability, and achieves substantially better performance than well-established baselines. Specifically, when training 7B-scale models for 60K steps, SimpleGPT achieves a training loss that is 0.08 lower than that of LLaMA2 with QKNorm, reducing the loss from 2.290 to 2.208. Our source code will be released at https://github.com/Ocram7/SimpleGPT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
