发表机构
University of New South Wales; University of Texas at Austin; City University of Hong Kong（新南威尔士大学; 德克萨斯大学奥斯汀分校; 香港城市大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文从Birkhoff几何角度分析流形约束超连接（mHC），揭示其双通道动力学、Sinkhorn作为回缩的几何性质，并证明梯度流与镜像下降的收敛速率差异。
AI 中文摘要
超连接将Transformer的残差流拓宽为n条并行流。其流形约束版本（mHC）在每一层用双随机矩阵混合这些流，该矩阵通过对指数化logits进行Sinkhorn归一化计算。我们在Birkhoff多面体上给出了这种设计的几何理论。首先，双随机混合器将流分成均值通道和差分通道，在均值通道上mHC恰好是残差网络，而差分通道每层以其第二奇异值σ2≤1-n min_{ij} H_{ij}收缩。因此，额外宽度是一种具有1/(1-σ2)层视界的衰减记忆，且在非负混合器中只有置换不会坍缩。其次，Sinkhorn-logit映射是一个全局坐标图，其logit梯度恰好是Fisher-Rao梯度。因此，logit梯度流遵循平方Fisher-Rao度量，而直通更新恰好是熵镜像下降。第三，在logit梯度流下，每个条目的对数以至多4n^3‖∇f‖∞ε的速率移动，其中ε是到最近置换的距离。因此，梯度流仅以1/t的速率接近和离开顶点，但镜像下降以指数速率移动。第四，Sinkhorn的局部收敛因子为σ2^2，因此固定的迭代预算限制了视界。实验证实了预测的速率。
英文摘要
Hyper-connections widen the residual stream of a Transformer to $n$ parallel streams. Their manifold-constrained version (mHC) mixes the streams at each layer with a doubly stochastic matrix, which it computes by Sinkhorn normalization of exponentiated logits. We give a geometric theory of this design on the Birkhoff polytope. First, a doubly stochastic mixer splits the stream into a mean channel, on which mHC is exactly a residual network, and a difference channel, which each layer contracts by its second singular value $σ_2 \le 1 - n \min_{ij} H_{ij}$. Thus the extra width is a fading memory with a horizon of $1/(1-σ_2)$ layers, and among nonnegative mixers only the permutations do not collapse. Second, the Sinkhorn-logit map is a global chart, and its logit gradient is exactly the Fisher-Rao gradient. Thus logit gradient flow follows a squared Fisher-Rao metric, and the straight-through update is exactly entropic mirror descent. Third, under logit gradient flow the logarithm of each entry moves at a rate of at most $4n^3\|\nabla f\|_\infty \varepsilon$, where $\varepsilon$ is the distance to the nearest permutation. Thus gradient flow approaches and leaves the vertices only at rate $1/t$, but mirror descent moves at an exponential rate. Fourth, the local convergence factor of Sinkhorn is $σ_2^2$, so a fixed iteration budget limits the horizon. Experiments confirm the predicted rates.
