无需牺牲的可解释性：混合解码器的忠实密集层分解
Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders
AI总结：
本文提出混合解码器（MxDs）通过层级稀疏性实现密集层分解，保持原始解码器的表达能力，显著提升稀疏性-准确性前沿性能。
摘要：
多层感知机（MLPs）是大型语言模型的重要组成部分，然而其密集表示使它们难以理解和编辑。最近的方法通过神经元层面的稀疏性学习可解释的近似，但无法忠实重建原始映射——显著增加模型的下一个标记交叉熵损失。在本文中，我们倡导转向层级稀疏性以克服稀疏层近似中的准确性权衡。在此范式下，我们引入了混合解码器（MxDs）。MxDs扩展了MLPs和门控线性单元，将预训练的密集层扩展成成千上万个专用子层。通过灵活的张量分解形式，每个稀疏激活的MxD子层实现一个具有满秩权重的线性变换——即使在高稀疏性下也能保持原始解码器的表达能力。实验表明，MxDs在具有多达3B参数的语言模型中，在稀疏性-准确性前沿上显著优于最先进的方法（例如Transcoders）。进一步的稀疏探测和特征引导评估显示，MxDs学习了与自然语言相似的专用特征——为设计可解释且忠实的分解开辟了新的前景。我们的代码包含在：https://github.com/james-oldfield/MxD/。
英文摘要：
Multilayer perceptrons (MLPs) are an integral part of large language models, yet their dense representations render them difficult to understand, edit, and steer. Recent methods learn interpretable approximations via neuron-level sparsity, yet fail to faithfully reconstruct the original mapping--significantly increasing model's next-token cross-entropy loss. In this paper, we advocate for moving to layer-level sparsity to overcome the accuracy trade-off in sparse layer approximation. Under this paradigm, we introduce Mixture of Decoders (MxDs). MxDs generalize MLPs and Gated Linear Units, expanding pre-trained dense layers into tens of thousands of specialized sublayers. Through a flexible form of tensor factorization, each sparsely activating MxD sublayer implements a linear transformation with full-rank weights--preserving the original decoders' expressive capacity even under heavy sparsity. Experimentally, we show that MxDs significantly outperform state-of-the-art methods (e.g., Transcoders) on the sparsity-accuracy frontier in language models with up to 3B parameters. Further evaluations on sparse probing and feature steering demonstrate that MxDs learn similarly specialized features of natural language--opening up a promising new avenue for designing interpretable yet faithful decompositions. Our code is included at: https://github.com/james-oldfield/MxD/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
