自适应优化器中掩码更新的意外有效性
On Surprising Effectiveness of Masking Updates in Adaptive Optimizers
AI总结：
Magma通过动量对齐梯度掩码提升自适应优化器性能，显著降低大型语言模型的困惑度。
摘要：
训练大型语言模型（LLMs）几乎完全依赖于密集的自适应优化器，且预条件器越来越复杂。我们通过展示随机掩码参数更新可以高度有效来挑战这一观点，其中RMSProp的掩码变体始终优于最近最先进的优化器。我们的分析表明，随机掩码诱导了一种依赖曲率的几何正则化，使优化轨迹更加平滑。受此发现启发，我们引入了动量对齐梯度掩码（Magma），它通过动量-梯度对齐来调节掩码更新。广泛的LLM预训练实验表明，Magma是自适应优化器的简单替代方案，具有稳定的收益和可忽略的计算开销。值得注意的是，对于1B模型规模，Magma相比Adam和Muon分别将困惑度降低了超过19%和9%。
英文摘要：
Training large language models (LLMs) relies almost exclusively on dense adaptive optimizers with increasingly sophisticated preconditioners. We challenge this by showing that randomly masking parameter updates can be highly effective, with a masked variant of RMSProp consistently outperforming recent state-of-the-art optimizers. Our analysis reveals that the random masking induces a curvature-dependent geometric regularization that smooths the optimization trajectory. Motivated by this finding, we introduce Momentum-aligned gradient masking (Magma), which modulates the masked updates using momentum-gradient alignment. Extensive LLM pre-training experiments show that Magma is a simple drop-in replacement for adaptive optimizers with consistent gains and negligible computational overhead. Notably, for the 1B model size, Magma reduces perplexity by over 19\% and 9\% compared to Adam and Muon, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
