Transformer中基于几何引导的分层前馈网络宽度分配

中文摘要 AI 辅助

前馈网络（FFN）占Transformer参数的很大一部分，但其隐藏层宽度通常在整个深度维度上保持恒定。本文探究是否可通过前向传播中对层行为的测量来分配该容量。我们将每个FFN视为传输词元表示点云的载体，使用对应保持偏移、Gromov-Wasserstein失真以及原始和尺度归一化度量下的一次持续同调来量化所诱导的几何变化。分层近似替代模型可生成精确的固定预算优化器。在七个预训练语言模型上的实验显示，原始欧氏工作量大致与残差范数增长同步，而归一化工作量主要集中在前端；Gromov-Wasserstein工作量与基于扰动的层敏感性的关联，比有限样本拓扑估计更为一致。在配对的128M和256M训练运行中，多种归一化工作量调度方案相比均匀宽度和手动设计的余弦衰减，均降低了平均验证损失；在放大的配对差异达到440M时，最佳的几何引导分配相比均匀宽度的提升幅度远大于余弦衰减，而反拓扑的原始对照组表现差于均匀宽度。

英文摘要

Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth. We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior. We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics. A layerwise approximation surrogate yields an exact fixed-budget optimizer. Across seven pretrained language models, raw Euclidean work largely tracks residual-norm growth, whereas normalized work is predominantly front-loaded. Gromov-Wasserstein work is more consistently associated with perturbation-based layer sensitivity than the finite-sample topological estimate. In paired 128M and 256M training runs, several normalized-work schedules reduce mean validation loss relative to both uniform width and a hand-designed cosine taper. With the amplified paired differences at 440M, the best geometry-based allocations improve over uniform substantially larger than the cosine taper, while the anti-topological raw control is worse than uniform.

发表机构

• University of Luxembourg（卢森堡大学）
• London Institute of Mathematical Sciences（伦敦数学科学研究所）

机构由 AI 辅助整理，请以论文原文为准。