arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.02309cs.LGcs.CL
收藏

go-$m$HC:通过广义正交随机矩阵直接参数化流形约束超连接

go-$m$HC: Direct Parameterization of Manifold-Constrained Hyper-Connections via Generalized Orthostochastic Matrices

Torque Dandachi, Sophia Diggs-Galligan

AI总结:

go-$m$HC通过广义正交随机矩阵直接参数化流形约束超连接,实现高效且精确的双重随机矩阵参数化,提升模型表达能力和效率。

摘要:

双重随机矩阵使学习的残差流混合得以实现,但精确且高效地参数化双重随机矩阵集(Birkhoff多面体)仍是一个开放性挑战。现有精确方法随流数量(d)呈阶数增长,而克罗内克分解方法效率高但表达能力有限。我们引入一种基于广义正交随机矩阵理论的新型精确参数化方法,其复杂度为$\mathcal{O}(d^3)$,暴露一个单个超参数s,连续插值于计算高效边界和完全表达的Birkhoff多面体之间。基于流形约束超连接(mHC)框架,我们在此参数化中实现了go-$m$HC。我们的方法与克罗内克分解方法自然组合,在相似FLOP成本下显著恢复了表达能力。谱分析表明,go-$m$HC比克罗内克分解基线更完整地填充了Birkhoff多面体。在合成流混合任务中,go-$m$HC实现最小理论损失并收敛速度达10倍以上。我们在3000万参数GPT风格语言模型中验证了我们的方法。go-$m$HC的表达能力、效率和精确性为将d作为模型容量的新维度提供了实用途径。

英文摘要:

Doubly stochastic matrices enable learned mixing across residual streams, but parameterizing the set of doubly stochastic matrices (the Birkhoff polytope) exactly and efficiently remains an open challenge. Existing exact methods scale factorially with the number of streams ($d$), while Kronecker-factorized approaches are efficient but expressivity-limited. We introduce a novel exact parameterization grounded in the theory of generalized orthostochastic matrices, which scales as $\mathcal{O}(d^3)$ and exposes a single hyperparameter $s$ which continuously interpolates between a computationally efficient boundary and the fully expressive Birkhoff polytope. Building on Manifold-Constrained Hyper-Connections ($m$HC), a framework for learned dynamic layer connectivity, we instantiate this parameterization in go-$m$HC. Our method composes naturally with Kronecker-factorized methods, substantially recovering expressivity at similar FLOP costs. Spectral analysis indicates that go-$m$HC fills the Birkhoff polytope far more completely than Kronecker-factorized baselines. On synthetic stream-mixing tasks, go-$m$HC achieves the minimum theoretical loss while converging up to $10\times$ faster. We validate our approach in a 30M parameter GPT-style language model. The expressivity, efficiency, and exactness of go-$m$HC offer a practical avenue for scaling $d$ as a new dimension of model capacity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑