HTMuon:通过重尾谱校正改进Muon
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
AI总结:
针对Muon优化器正交化更新规则抑制重尾权重谱并过度强调噪声方向训练的问题,提出HTMuon方法,通过重尾自正则化理论产生更重尾的更新和权重谱,在LLM预训练和图像分类中提升性能。
摘要:
Muon最近在LLM训练中显示出有希望的结果。在这项工作中,我们研究如何进一步改进Muon。我们认为Muon的正交化更新规则抑制了重尾权重谱的出现,并过度强调了沿噪声主导方向的训练。受重尾自正则化(HT-SR)理论的启发,我们提出了HTMuon。HTMuon保留了Muon捕捉参数相互依赖性的能力,同时产生更重尾的更新并诱导更重尾的权重谱。在LLM预训练和图像分类上的实验表明,HTMuon持续优于最先进的基线,并且可以作为现有Muon变体的插件使用。例如,在C4数据集上的LLaMA预训练中,与Muon相比,HTMuon将困惑度降低了高达0.98。我们进一步从理论上证明,HTMuon对应于Schatten-$q$范数约束下的最速下降,并提供了在光滑非凸环境下的收敛性分析。HTMuon的实现可在https://github.com/TDCSZ327/HTmuon获取。
英文摘要:
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon's orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by the Heavy-Tailed Self-Regularization (HT-SR) theory, we propose HTMuon. HTMuon preserves Muon's ability to capture parameter interdependencies while producing heavier-tailed updates and inducing heavier-tailed weight spectra. Experiments on LLM pretraining and image classification show that HTMuon consistently improves performance over state-of-the-art baselines and can also serve as a plug-in on top of existing Muon variants. For example, on LLaMA pretraining on the C4 dataset, HTMuon reduces perplexity by up to $0.98$ compared to Muon. We further theoretically show that HTMuon corresponds to steepest descent under the Schatten-$q$ norm constraint and provide convergence analysis in smooth non-convex settings. The implementation of HTMuon is available at https://github.com/TDCSZ327/HTmuon.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。