arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.03459cs.LGcs.CL
收藏

Transformer 中的 MLP 非线性浪费问题:测量与重新分配

Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer's MLP Budget

Peter Balogh

AI总结:

研究发现Transformer中MLP非线性浪费问题,通过门控技术将部分MLP替换为线性层,从而降低困惑度并提升性能。

摘要:

我们研究了Transformer中的MLP非线性何时实际上是必要的。一个具有d+1个参数的门控决定何时用线性替代品替换完整的MLP。通过在六个模型(162M-2.8B参数)、两种架构和三种语料库上系统研究,我们发现非线性需求不能从token身份预测:跨语料库相关性为零(r < 0.05)。路由决策完全依赖于上下文。尽管单个实例的可预测性较弱,但门控利用了一个高度偏斜的分布,其中大多数MLP计算接近线性,从而在GPT-2中以<1%的困惑度成本实现了25-56%的线性路由。在GPT-2 Large中,11个36层中的层通过门控优于基线,且无层超过3.7%的全线性成本。这一成功是架构依赖的:Pythia模型显示了更高的成本,尽管Pythia-2.8B的完整32层扫描揭示了一个层,其性能略微优于基线。作为概念验证,我们逐步用冻结的线性矩阵替换中间层MLP:24个层中有5个在零成本下线性化。使用完整训练预算,4个线性化层实现了10.2%的困惑度改进——而双阶段门控方法将其推高到17.3%,优于普通微调控制,证实了这些层中的非线性MLP实际上是有害的。

英文摘要:

We investigate when transformer MLP nonlinearity is actually necessary. A gate with $d+1$ parameters decides when to replace the full MLP with a linear surrogate. Through systematic investigation across six models (162M-2.8B parameters), two architectures, and three corpora, we establish that nonlinearity need cannot be predicted from token identity: cross-corpus correlation is zero ($r < 0.05$). The routing decision is fully contextual. Despite weak per-instance predictability, the gate exploits a heavily skewed distribution where most MLP computations are near-linear, achieving 25-56% linear routing at <1% perplexity cost in GPT-2. In GPT-2 Large, 11 of 36 layers beat baseline with gating and no layer exceeds 3.7% all-linear cost. This success is architecture-dependent: Pythia models show higher costs, though Pythia-2.8B's full 32-layer sweep reveals one layer that narrowly beats baseline. As a proof of concept, we progressively replace middle-layer MLPs with frozen linear matrices: 5 of 24 layers linearize at zero cost. With a full training budget, 4 linearized layers yield a 10.2% perplexity improvement -- and a two-phase gated approach pushes this to 17.3%, beating a vanilla fine-tuning control and confirming that the nonlinear MLPs at these layers were actively harmful.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑