arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01105cs.LGcs.AI
收藏

OLion:通过谱和ℓ∞隐偏见的交集接近Hadamard理想

OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases

Peking University(北京大学);Microsoft Research Asia(微软亚洲研究院)

Zixiao Wang, Yifei Shen, Huishuai Zhang

AI总结:

OLion通过结合谱控制与ℓ∞隐偏见,高效近似在Hadamard-like集上取最大步长,提升大规模语言和视觉任务的优化性能。

摘要:

许多优化器可以被解释为在范数诱导几何下最陡下降法,因此继承了相应的隐偏见。我们引入\nameA{}(\fullname{}),它结合了从正交化更新方向获得的谱控制与从符号更新获得的ℓ∞风格坐标控制。\nameA{}形成一种Lion风格的动量方向,通过几次牛顿-施尔茨迭代近似正交化它,然后应用逐元素符号,提供了一种高效的近似方法,以在谱和ℓ∞约束集的交集(一种用于矩阵参数的缩放Hadamard-like集)上取最大步长。尽管正交化和符号的非线性很强,我们在一个温和且经实验证明的对角等方差假设下证明了收敛性。在大规模语言和视觉训练中,包括GPT-2和Llama预训练、SiT图像预训练以及监督微调中,\nameA{}在可比调参下匹配或优于AdamW和Muon,仅使用动量级优化器状态,并在微调AdamW预训练检查点时缓解了优化器不匹配问题。

英文摘要:

Many optimizers can be interpreted as steepest-descent methods under norm-induced geometries, and thus inherit corresponding implicit biases. We introduce \nameA{} (\fullname{}), which combines spectral control from orthogonalized update directions with $\ell_\infty$-style coordinate control from sign updates. \nameA{} forms a Lion-style momentum direction, approximately orthogonalizes it via a few Newton--Schulz iterations, and then applies an entrywise sign, providing an efficient approximation to taking a maximal step over the intersection of the spectral and $\ell_\infty$ constraint sets (a scaled Hadamard-like set for matrix parameters). Despite the strong nonlinearity of orthogonalization and sign, we prove convergence under a mild, empirically verified diagonal-isotropy assumption. Across large-scale language and vision training, including GPT-2 and Llama pretraining, SiT image pretraining, and supervised fine-tuning, \nameA{} matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level optimizer state, and it mitigates optimizer mismatch when fine-tuning AdamW-pretrained checkpoints.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑