发表机构
MIRIAI（MIRIAI研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对Muon优化器的符号压缩问题，分析不同符号放置方式及误差反馈的效果，发现神谕后取符号的启发式压缩方法在多任务实验中表现优于理论收敛的变体。
AI 中文摘要
SignMuon通过对Muon更新的每个参数取元素级符号，将其压缩至每个参数1比特，为在极低通信预算下运行感知矩阵的优化器提供了最直接的方式。它在实际表现上优于SignSGD，甚至能在线性函数上上升。在线性最小化神谕（LMO）前而非后对梯度取符号并不能修复该问题：我们构造了一个小型显式实例，其中在神谕前取符号（MuonUSign）和在神谕两侧取符号（MuonSign）同样会上升，因此神谕周围的符号放置总体上无法实现下降。误差反馈作为针对有偏压缩器的标准补救措施，无法拯救SignMuon：将其应用于Muon输出时，无论平滑度常数、步长和动量如何，误差反馈均会失效。而将误差反馈应用于梯度时则有效，EF21-MuonUSign和EF21-MuonSign在光滑非凸问题上的平方梯度范数达到了标准的$\tilde{O}(T^{-1/2})$速率，后者每方向仅需1比特。实验结果则反转了排序：在集中式CIFAR-10、联邦式CIFAR-10和nanoGPT速度测试中，表现最强的压缩方法始终是在神谕后取符号，即我们证明会发散的放置方式，而可证明收敛的变体性能落后。在这些规模下，神谕后压缩这一启发式方法比理论保证更重要。
英文摘要
SignMuon compresses the Muon update to one bit per parameter by taking its elementwise sign, providing the most direct way to run a matrix-aware optimizer under an extremely low communication budget. It outperforms SignSGD in practice, yet it can ascend even on a linear function. Signing the gradient before the Linear Minimization Oracle (LMO), rather than after, does not repair this: we construct a small explicit instance on which sign-before (MuonUSign) and sign-on-both-sides (MuonSign) ascend as well, so no placement of the sign around the oracle descends in general. Error feedback, the standard remedy for a biased compressor, does not rescue SignMuon: when applied to Muon's output, error feedback can fail for every smoothness constant, step size, and momentum. Applied to the gradient, error feedback does work, and EF21-MuonUSign and EF21-MuonSign attain the standard $\mathcal{O}(T^{-1/2})$ rate for the squared gradient norm on smooth nonconvex problems, the latter at one bit in each direction. Experiments then reverse the ordering: across centralized CIFAR-10, federated CIFAR-10, and the nanoGPT speedrun, the strongest compressed method is consistently sign-after-the-LMO, precisely the placement we prove divergent, with the provably convergent variants trailing it. Compressing after the LMO, a heuristic, matters more at these scales than the guarantee does.
Comments42 pages, 13 figures. Code: https://github.com/intsystems/signmuon
