稀疏门控的微型线性专家
Sparsely gated tiny linear experts
AI总结：
提出稀疏门控线性神经元（sgatlin）网络，通过将每个专家缩减为单个神经元并去除非线性，在等计算量下提升语言模型困惑度，同时增强可解释性。
摘要：
稀疏性允许在不按比例增加计算成本的情况下扩展模型参数。虽然混合专家（MoE）模型变得越来越稀疏，但单个专家通常仍然庞大且密集。在这里，我们通过将每个专家缩减为单个神经元并选择许多可用神经元中的极小一部分，进一步增加稀疏性，从而提高计算效率和可解释性。与直觉相反，实现这两者的关键是去除通常应用于专家的非线性，从而得到一个稀疏门控线性神经元（sgatlin）网络。在等计算量比较中，我们发现用sgatlin替换所有Transformer前馈层可以在不同计算预算下改善语言模型的困惑度。同时，由此产生的前馈电路的稀疏性和线性为模型可解释性提供了新的机会。在一个小规模案例研究中，我们证明sgatlin中的前馈电路可以在无需训练额外替代模型的情况下进行解释。我们发现它们形成了语义结构化的聚类，并且在因果上参与了事实回忆。我们的发现为计算高效且可解释的Transformer前馈层指明了一条可能的路径。
英文摘要：
Sparsity allows scaling model parameters without proportionally increasing computational cost. While mixture of experts (MoE) models are made increasingly sparse, individual experts typically remain large and dense. Here, we demonstrate that further increasing sparsity by shrinking each expert to consist of a single neuron and selecting a tiny fraction of many available neurons can improve compute efficiency and interpretability. Counterintuitively, the key to achieving both is removing the nonlinearity typically applied to the experts, resulting in a network of sparsely gated linear neurons (sgatlin). In an isoflop comparison, we find that replacing all transformer feedforward layers with sgatlin improves perplexity in language models across different compute budgets. At the same time, the sparsity and linearity of the resulting feedforward circuits present new opportunities for model interpretability. In a small-scale case study, we demonstrate that feedforward circuits in sgatlin can be interpreted without having to train additional replacement models. We find that they form semantically structured clusters and are causally implicated in factual recall. Our findings paint a possible path towards compute-efficient and interpretable transformer feedforward layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
