arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.20896cs.LGcs.AI
收藏

超越Birkhoff多面体:谱球约束超连接

Spectral-Sphere-Constrained Hyper-Connections

University of Maryland, College Park(马里兰大学 College Park 分校);University of Utah(犹他大学)

Zhaoyi Liu, Haichuan Zhang, Ang Li

AI总结:

本文提出Spectral-Sphere-Constrained Hyper-Connections (sHC),通过将可行集从刚性多面体转换为谱范数球,允许负数条目,从而解锁选择性特征多样化,解决传统Birkhoff多面体约束的三个局限。

摘要:

超连接(HC)将残差连接扩展为多流,利用残差矩阵进行跨流特征混合以增强模型表达能力。然而,无约束混合破坏了残差连接固有的身份映射属性,导致训练不稳定。为解决此问题,Manifold-Constrained Hyper-Connections (mHC) 及其变体通过Sinkhorn迭代或基于排列的参数化将这些矩阵限制在Birkhoff多面体(双随机矩阵)内。我们揭示了该多面体约束的三个局限:(1)身份退化,其中学习的矩阵塌陷在身份附近,削弱跨流交互;(2)表达性瓶颈,非负约束阻止了减法特征解耦;(3)参数化效率低下,表现为不稳定的Sinkhorn迭代或排列参数化的阶乘计算开销。为克服这些缺陷,我们提出Spectral-Sphere-Constrained Hyper-Connections (sHC)。通过将可行集从刚性多面体几何地转换为谱范数球,sHC允许负数条目,解锁选择性特征多样化。此转换消除了不稳定Sinkhorn投影和阶乘参数化,使表达性强、非退化的残差矩阵得以保持训练稳定性。

英文摘要:

Hyper-Connections (HC) extend residual connections into multiple streams, employing residual matrices for cross-stream mixing to enrich model expressivity. However, unconstrained mixing disrupts the identity mapping property intrinsic to the residual connection, causing unstable training. To address this, Manifold-Constrained Hyper-Connections (mHC) and its variants restrict these matrices to be doubly stochastic via Sinkhorn-Knopp (SK) algorithm or permutation-based parameterizations. We reveal three limitations of this doubly stochastic constraint: (1) identity degeneration, where learned matrices collapse around the identity initialization and diminish cross-stream interactions, (2) a expressivity bottleneck, where the doubly stochastic constraint restricts the freedom of the subdominant spectrum of the residual matrices, preventing the model from selectively preserving or attenuating cross-stream variations, and (3) parameterization inefficiencies, manifesting as unstable SK iterations or the factorial-scaling overhead of permutation-based parameterizations. To overcome these flaws, we propose Spectral-Sphere-Constrained Hyper-Connections ($\mathrm{s}^{2}$HC). By confining residual matrices to a spectral norm sphere, $\mathrm{s}^{2}$HC restores free control over the subdominant spectrum, enabling the model to selectively preserve or attenuate cross-stream variations. This shift eliminates unstable SK iterations and factorial parameterization, enabling expressive, non-degenerate residual matrices while preserving training stability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑