缪子：超越归一化动量的μ子表示几何

中文摘要 AI 辅助

μ子风格的优化器将极坐标映射应用于矩阵动量，但其更新也取决于正交化前每个参数块的表示。我们将这种表示选择作为优化器几何的一种形式进行研究，并引入了{\方法}，这是一族μ子风格的优化器，在原生、近方形、瘦长和向量表示中共享相同的动量规则和牛顿 - 舒尔茨后端。每种弗罗贝尼乌斯等距表示都诱导出一种独特的极坐标最速下降几何，其中较短的矩阵维度决定了支持的奇异通道数量、回拉缩放以及随机非凸收敛界中的常数。在师生模型中，曲率坍塌和各向同性的马尔琴科 - 帕斯特谱轮廓将早期耗散与表示的核范数与弗罗贝尼乌斯平方范数之比联系起来。对LLaMA2 - 130M和LLaMA2 - 600M的预训练实验以及固定动量诊断表明，平衡的非原生表示可以匹配原生表示的性能，而减小较短维度会削弱缩放和奇异通道支持，导致行为越来越类似于归一化动量。

英文摘要

Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer geometry and introduce {\method}, a family of Muon-style optimizers that shares the same momentum rule and Newton--Schulz backend across native, nearest-square, skinny, and vector representations. Each Frobenius-isometric representation induces a distinct polar steepest-descent geometry, in which the shorter matrix dimension determines the number of supported singular channels, the pullback scaling, and the constants in stochastic nonconvex convergence bounds. In a teacher--student model, curvature collapse and an isotropic Marchenko--Pastur spectral profile connect early-stage dissipation to the represented nuclear-to-squared-Frobenius norm ratio. Pretraining experiments on LLaMA2-130M and LLaMA2-600M, together with fixed-momentum diagnostics, show that balanced non-native representations can match the performance of the native representation, whereas reducing the shorter dimension weakens the scaling and singular-channel support, leading to behavior that increasingly resembles normalized momentum.

发表机构

• Pengcheng Laboratory（鹏城实验室）
• Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
• Southern University of Science and Technology（南方科技大学）
• Shenzhen University of Advanced Technology（深圳先进技术大学）
• University of Chinese Academy of Sciences（中国科学院大学）

机构由 AI 辅助整理，请以论文原文为准。