发表机构
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对物理神经网络训练中优化器物理可实现性与性能的权衡，提出物理Muon，将正交化作为连续时间流的平衡计算，用矩阵-向量乘积近似，在10.95M参数Transformer上验证了与Newton-Schulz相当的训练性能。
AI 中文摘要
物理神经网络和模拟内存计算有望降低神经网络训练的能量成本。然而，要实现这一潜力，需要将有效学习与物理可实现性相结合的优化器。SGD适合局部模拟更新，但在Transformer上表现不佳，而Adam系列对模拟偏差不稳定。Muon提供了强大的训练性能，但其Newton-Schulz正交化依赖于稠密矩阵-矩阵乘积。为了解决这一障碍，我们引入了物理Muon，它将正交化计算为连续时间流的平衡。随机探针使用矩阵-向量乘积、倒数读取和局部秩-1写入来近似该流。为了测试这种替换是否保持训练性能，我们在一个10.95M参数的Transformer上进行了评估。稠密流的平均验证交叉熵在每种方法九个种子上比Newton-Schulz高0.0085；探针实现在两个种子上比对照组高0.0188。电路仿真进一步再现了流动力学，并产生了可比较的训练行为。
英文摘要
Physical neural networks and analog in-memory computing could reduce the energy cost of neural network training. Realizing this potential, however, requires optimizers that combine effective learning with physical implementability. SGD fits local analog updates but struggles on transformers, while Adam family is unstable against analog bias. Muon offers strong training performance, but its Newton--Schulz orthogonalization relies on dense matrix-matrix products. To address this obstacle, we introduce Physical Muon, which computes the orthogonalization as the equilibrium of a continuous-time flow. Random probes approximate the flow using matrix-vector products, reciprocal reads, and local rank-1 writes. To test whether this replacement preserves training performance, we evaluate it on a 10.95M-parameter transformer. The dense flow's mean validation cross-entropy is 0.0085 above Newton--Schulz across nine seeds per method; the probe implementation is 0.0188 above the control across two seeds. Circuit simulations further reproduce the flow dynamics and yield comparable training behavior.
CommentsAccepted at 18th annual workshop on Optimization for Machine Learning
