发表机构
Columbia University（哥伦比亚大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究将语言模型头部与softmax作为整体推导更新几何，基于Muon的奇异值条件提出最大化最小行间距并约束直径的方法，为V≫d时的近似等距问题提供了思路。
AI 中文摘要
我们将语言模型头部与softmax作为单个模块研究，从二者的组合而非孤立的权重矩阵中推导更新几何。在希尔伯特投影距离下，当||h||₂≤H时，更新S导致的最大变化为H乘以其token行的欧氏直径，即H乘以maxᵢ<j||sᵢ-sⱼ||₂。受Muon的奇异值条件启发，我们提出在约束该直径的同时最大化最小行间距，当V≫d时会产生近似等距问题。
英文摘要
Muon motivates designing optimizer geometry around the function of each parameter block and uses the spectral norm for hidden linear layers. For the language-model head, the spectral norm is not a faithful measure of functional change. Softmax removes shared logit shifts, whereas the spectral norm can assign arbitrarily large size to updates that change no output probability. We therefore treat the LM head and softmax as one module and derive an update geometry for their composition. Hilbert's projective distance respects this invariance as it measures the largest change in pairwise log odds. For an update $S$ with token rows $s_i^\top$, we show that the largest Hilbert distance over $\left\lVert h\right\rVert_2\leq H$ is exactly $H D(S)$, where $D(S)=\max_{i<j}\left\lVert s_i - s_j\right\rVert_2$ is the Euclidean row diameter. This diameter replaces the spectral norm in the resulting Muon-style steepest descent problem. An exact solution is possible, but its direct formulation contains one $d$-dimensional vector variable for every token pair. For a vocabulary size of approximately $50$k, this means more than one billion token pairs, making the calculation impractical at every training step. We instead impose a stronger common-ball constraint and derive projected RowNorm as an $O(Vd)$ solution. For the exact RowNorm oracle, we prove that its first-order decrease is at least $1/\sqrt{2}$ of the exact diameter-constrained optimum. With Muon on the backbone, experiments across three seeds at 190M, 380M, and 640M parameters show that RowNorm reduces mean final step diameters and empirical Hilbert RMS perturbations by factors of $45$--$60$ and $12$--$15$, respectively, with only a $0.0057$--$0.0153$ increase in mean final validation loss.
