PolyGLU：Transformer前馈网络中的状态条件激活路由
PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks
AI总结：
PolyGLU通过动态路由机制使FFN神经元可选择四种激活函数，提升模型灵活性和性能，实验表明其在标准基准测试中表现优异。
摘要：
生物神经系统利用多种神经递质如谷氨酸、GABA、多巴胺和乙酰胆碱在共享神经回路中实现不同的信号处理模式。相比之下，现代Transformer在所有前馈神经元中使用单一固定激活函数。我们引入PolyGLU（多色门控线性单元），作为SwiGLU的替代方案，使每个FFN神经元能够通过可微机制动态路由至K=4种激活函数，结合学习的静态偏好与输入条件门控，通过Gumbel-Softmax端到端训练。我们训练了具有597000000个参数的Transformer模型PolychromaticLM，在约1000亿个token上使用单块NVIDIA A100 GPU进行训练。我们的关键发现是涌现的路由行为：在不使用任何显式稀疏损失或熵正则化的情况下，路由机制收敛到接近确定性的激活选择（平均动态熵为最大值的0.030%），呈现出深度依赖的专业化模式——早期层偏好GELU，深层层强烈偏好Tanh。三个层保持较高的路由熵，表明计算灵活性点。路由架构仅增加0.23%的参数开销（约1400000个参数），并证明在监督微调中完全鲁棒：路由熵在整个13067次SFT步骤中保持不变。在标准基准测试中，PolychromaticLM在训练token数量少3600倍的情况下，达到Qwen3-0.6B-Base性能的62-89%。所有代码、权重和训练基础设施均在Apache 2.0下发布。
英文摘要：
Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
