发表机构
The University of Manchester; Zhejiang University（曼彻斯特大学; 浙江大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文揭示tied-β Adam中隐藏的近似尺度稳定比率结构，提出基于4位码本的压缩方法，并阐明其与符号方法的联系，实现高效且性能相当的优化。
AI 中文摘要
Adam是训练现代深度神经网络的默认优化器，然而由于其一阶和二阶矩指数移动平均（EMA）之间的复杂交互，其自适应行为仍未被充分理解。我们在tied-$\eta$机制下研究Adam，即两个EMA衰减率相等时，并表明其自适应动力学可以通过一个变换后的比率来表达，该比率具有近似尺度稳定的行为。经验上，这个变换后的比率在不同任务、模型规模和训练阶段中表现出稳定的重尾分布，这与原始矩量值的变异性形成对比。这种经验稳定性具有实践和概念上的双重意义。首先，我们推导出变换后比率的递推关系，从而得到Adam的一种重新参数化，用可压缩状态替代二阶矩。利用其稳定分布，我们表明在实验中，一个固定的4位码本足以存储该状态而无需辅助缩放，实现了与全精度Adam相媲美的性能。其次，变换后比率的视角阐明了Adam与基于符号的方法的联系：Adam简化为由变换后比率调制的基于符号的动量，而将其替换为常数则恢复Signum作为极限情况。这一视角进一步提供了两种方法之间转移学习率的简单规则。总之，这些结果表明，tied-$\eta$ Adam在其自适应行为之下具有一个简单且近似稳定的比率结构，并展示了其在分析和高效实现方面的实用性。
英文摘要
Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-$β$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw moment magnitudes. This empirical stability has both practical and conceptual consequences. First, we derive a recurrence for the transformed ratio, yielding a reparameterization of Adam that replaces the second moment with a compressible state. Leveraging its stable distribution, we show that a fixed 4-bit codebook is sufficient in our experiments to store this state without auxiliary scaling, achieving performance competitive with full-precision Adam. Second, the transformed ratio view clarifies Adam's connection to sign-based methods: Adam reduces to sign-based momentum modulated by the transformed ratio, and replacing it with a constant recovers Signum as a limiting case. This perspective further provides a simple rule for transferring learning rates between the two methods. Together, these results suggest that tied-$β$ Adam admits a simple and approximately stable ratio structure underlying its adaptive behavior and demonstrate its utility for both analysis and efficient implementation.
