论离散表示在稀疏混合专家中的作用
On the Role of Discrete Representation in Sparse Mixture of Experts
AI总结：
针对 SMoE 路由器的不一致与表示坍塌问题，本文提出基于向量量化离散表示间接分配专家的 VQMoE，并在语言和视觉任务中验证其鲁棒性提升 28%。
摘要：
稀疏混合专家（SMoE）是在不增加计算成本的情况下扩展模型容量的有效解决方案。SMoE 的一个关键组件是路由器，负责将输入引导至相关专家；然而，它也带来了主要弱点，导致路由不一致和表示坍塌问题。不同于以往工作修复路由器的做法，我们提出一种替代方案，通过间接方式为输入分配专家，该方案采用指向专家的输入离散表示。离散表示通过向量量化学习，由此产生名为 Vector-Quantized Mixture of Experts（VQMoE）的新架构。我们提供理论支持和实证证据，证明 VQMoE 能够克服传统路由器中存在的挑战。通过在大语言模型和视觉任务的预训练与微调上进行广泛评估，我们表明 VQMoE 相比其他 SMoE 路由方法在鲁棒性上实现 28% 的提升，同时在微调任务中保持强劲性能。
英文摘要：
Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to relevant experts; however, it also presents a major weakness, leading to routing inconsistencies and representation collapse issues. Instead of fixing the router like previous works, we propose an alternative that assigns experts to input via indirection, which employs the discrete representation of input that points to the expert. The discrete representations are learnt via vector quantization, resulting in a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE). We provide theoretical support and empirical evidence demonstrating the VQMoE's ability to overcome the challenges present in traditional routers. Through extensive evaluations on both large language models and vision tasks for pre-training and fine-tuning, we show that VQMoE achieves a 28% improvement in robustness compared to other SMoE routing methods, while maintaining strong performance in fine-tuning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
