NeuronMoE：基于神经元的混合专家架构用于高效多语言大语言模型扩展
NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
AI总结：
NeuronMoE通过分析神经元多样性优化专家分配，实现低资源语言模型的高效扩展，减少40%参数并保持性能。
摘要：
将大型语言模型扩展到低资源语言对于全球可访问性至关重要，但为每种语言训练单独的模型成本过高。混合专家（MoE）架构通过添加稀疏的语言特定参数来解决这一问题，但确定每层需要多少专家仍是一个开放性问题。当前方法基于层级相似性分配专家，但语言处理在单个神经元层面表现出细粒度的专业化。我们提出了NeuronMoE，一种分析所有Transformer组件中语言特定神经元的方法，以指导每层的专家分配，基于经验测量的跨语言神经元多样性。应用于Llama-3.2-3B用于低资源语言（希腊语、土耳其语和匈牙利语）时，该方法在平均参数减少约40%的同时，与LayerMoE基线保持性能相当。我们发现低资源语言专家独立发展出与高资源语言相似的神经元专业化模式，这些模式集中在早期和晚期层。这揭示了多语言模型如何组织语言知识的潜在通用架构原则。
英文摘要：
Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
