arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.17285cs.LG
收藏

哪一层采用哪种几何?混合优化器训练中的可测量符号-谱分配

Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training

Stanford University(斯坦福大学);Mila(米拉研究所);Université de Montréal(蒙特利尔大学);Arizona State University(亚利桑那州立大学);Nanyang Technological University(南洋理工大学);SB Intuitions Corp.(SB直觉公司)

Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi, Ganesh Talluri, Atsushi Nitanda, Seng Pei Liew, Ryuichiro Hataya, Ioannis Mitliagkas

AI总结:

该研究将神经网络训练的符号与谱更新分配转化为逐层可测量决策,提出G-Scion方法控制层路由,实验显示其可优化GPT-2、ViT-B/16等模型的验证损失,表明优化器几何无需按层类型硬编码。

摘要:

现代神经网络训练采用混合方案,将类AdamW或基于符号的更新分配给嵌入层和输出层,将谱更新分配给隐藏矩阵。这些分配通常按层类型固定,缺乏选择合适几何的明确准则。我们将符号与谱的分配转化为可测量的逐层决策问题。对于理想无记忆步,我们推导了谱更新与符号更新的保守下降引理下界的有限批处理比率,其闭式形式将信号效率与依赖几何的全局曲率因子分离。在正信号 regime 中,相同的平滑性假设给出双侧界,当比率穿过显式噪声相关阈值时,可确定哪一步具有更大的条件期望下降。随后,我们从梯度对齐和海森向量积中推导了局部方向二次得分,并将其用于Generalized-Scion(G-Scion)以控制逐层路由。在受控输入秩实验中,包含方向曲率的方法在所有7种聚合设置中匹配了测量的同迭代排序,而无曲率代理则选择了相反的几何。在31M GPT-2上,输出得分将输出层从谱预热路由至符号,且G-Scion在无记忆和匹配动量比较中,相比固定Scion提升了验证损失。ViT-B/16结果进一步表明,谱边缘分配在语言建模之外可优于固定Scion。这些结果表明,优化器几何无需按层类型硬编码,是理解应如何分配优化器几何的第一步。

英文摘要:

Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignments are usually fixed by layer type, without a clear criterion for selecting the appropriate geometry. We turn sign-versus-spectral assignment into a measurable, layerwise decision problem. For ideal memoryless steps, we derive a finite-batch ratio of conservative descent-lemma lower bounds for spectral and sign updates. Its closed form separates signal efficiency from a geometry-dependent global curvature factor. In the positive-signal regime, the same smoothness assumption gives two-sided bounds that determine which step has the larger conditional expected decrease when the ratio crosses explicit noise-dependent thresholds. We then derive a local directional quadratic score from gradient alignments and Hessian-vector products and use it in Generalized-Scion (G-Scion) to control layerwise routing. In a controlled input-rank experiment, including directional curvature matches the measured same-iterate ordering in all seven aggregate settings, whereas the curvature-free proxy selects the opposite geometry. On 31M GPT-2, the output score routes the output layer from spectral warmup to sign, and G-Scion improves validation loss over fixed Scion in memoryless and matched-momentum comparisons. ViT-B/16 results further show that a spectral edge assignment can outperform fixed Scion outside language modelling. These results suggest that optimizer geometry need not be hard-coded by layer type and are a first step toward understanding how it should be assigned.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑