面向Muon及同类算法的误差反馈
Error Feedback for Muon and Friends
AI总结：
针对Muon等非欧几里得LMO类优化器缺乏原则性分布式框架、存在通信瓶颈的问题，提出首个带严格收敛保证的通信高效优化器EF21-Muon，实现误差反馈向非欧场景的拓展，实验显示其最高可节省7倍通信量且精度无损。
摘要：
近期的优化器如Muon、Scion和Gluon通过在非欧几里得范数球上利用逐层线性最小化预言机（LMOs），以传统算法无法实现的方式捕捉神经网络结构，推动了大规模深度学习的前沿发展。然而，目前尚无针对这些方法的原则性分布式框架，通信瓶颈问题也未得到解决。仅有的少数分布式变体属于启发式方法，尚无收敛保证。我们提出了EF21-Muon，这是首个具备严格收敛保证的通信高效、基于非欧几里得LMO的优化器。EF21-Muon支持随机梯度、动量以及带误差反馈的双向压缩，标志着误差反馈首次拓展至非欧几里得场景之外。当关闭压缩并选择特定范数时，它可还原为Muon/Scion/Gluon，为这一强大的优化器家族提供了首个高效的分布式实现。我们的理论覆盖了非欧几里得光滑场景以及更通用的$(L^0, L^1)$-光滑场景，匹配了已知最优的欧几里得收敛速率，且在合适的范数选择下可实现更快收敛。我们进一步将分析拓展至逐层（广义）光滑机制，以捕捉深度网络的各向异性结构。在NanoGPT基准上的实验将EF21-Muon与未压缩的Muon/Scion/Gluon进行对比，结果表明其可实现最高7倍的通信节省，且无精度下降。
英文摘要：
Recent optimizers like Muon, Scion, and Gluon have pushed the frontier of large-scale deep learning by exploiting layer-wise linear minimization oracles (LMOs) over non-Euclidean norm balls, capturing neural network structure in ways traditional algorithms cannot. Yet, no principled distributed framework exists for these methods, and communication bottlenecks remain unaddressed. The very few distributed variants are heuristic, with no convergence guarantees in sight. We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based optimizer with rigorous convergence guarantees. EF21-Muon supports stochastic gradients, momentum, and bidirectional compression with error feedback-marking the first extension of error feedback beyond the Euclidean setting. It recovers Muon/Scion/Gluon when compression is off and specific norms are chosen, providing the first efficient distributed implementation of this powerful family. Our theory covers non-Euclidean smooth and the more general $(L^0, L^1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices. We further extend the analysis to layer-wise (generalized) smoothness regimes, capturing the anisotropic structure of deep networks. Experiments on NanoGPT benchmarking EF21-Muon against uncompressed Muon/Scion/Gluon demonstrate up to $7\times$ communication savings with no accuracy degradation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
