SwiGLU的开放正尾是否必要？来自带MemGLU的闭尾门控的证据
Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU
浏览论文内容
中文总结 AI 辅助
该研究针对仅解码器的语言模型FFN，通过对比实验发现，源自忆阻分支几何的MemGLU在相近损失下无需SwiGLU的开放正尾即可表现良好，表明模型会适应预训练的门控几何。
中文摘要 AI 辅助
我们测试仅解码器的语言模型前馈网络（FFN）是否需要SwiGLU的开放正尾。我们引入MemGLU作为源自忆阻分支几何的闭尾对比模型。在3个随机种子下开展的9M和30M预训练配对运行中，MemGLU的验证负对数似然（NLL）始终与SwiGLU相差约0.1%。训练后的SwiGLU检查点对正尾抑制敏感，而机制诊断显示，尽管损失相近，两个模型的门控使用方式存在差异。这些结果表明，模型会适应预训练期间可用的门控几何，在测试规模下，SwiGLU的开放正尾对仅解码器的语言模型FFN并非必要。
英文摘要
We test whether decoder-only language-model FFNs require SwiGLU's open positive tail. We introduce MemGLU as a closed-tail comparator derived from a memristive branch geometry. Across paired 9M and 30M pretraining runs with three seeds, MemGLU remains within about 0.1% of SwiGLU in validation NLL. Trained SwiGLU checkpoints are sensitive to positive-tail suppression, while mechanism diagnostics show that the two models use their gates differently despite similar losses. These results suggest that models adapt to the gate geometry available during pretraining. At the tested scales, SwiGLU's open positive tail is not necessary for decoder-only language-model FFNs.
发表机构
- City University of Hong Kong（香港城市大学）
- Beihang University（北京航空航天大学）
- National University of Singapore（新加坡国立大学）
机构由 AI 辅助整理，请以论文原文为准。
