ViMoE：视觉混合专家模型设计的实证研究
ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts
AI总结：
本文将MoE结构整合到ViT中提出ViMoE，针对其性能对MoE层配置敏感的问题引入共享专家解决路由不可靠问题，并通过分析专家路由行为优化模型，为视觉MoE设计提供实证指导。
摘要：
混合专家模型（Mixture-of-Experts，MoE）体现了分治思想，是提升模型容量的有效方法，在多领域展现出优异的可扩展性。本文将MoE结构整合到经典视觉Transformer（Vision Transformer，ViT）中，命名为ViMoE，并通过图像分类和语义分割任务的全面研究探索MoE在视觉领域的应用潜力。然而，我们发现模型性能对MoE层的配置高度敏感，若不精心设计难以获得最优结果。其根本原因是不当的MoE层会导致路由不可靠，阻碍专家有效获取有用信息。为此，我们引入共享专家来学习和捕捉通用知识，这是构建稳定ViMoE的有效方法。此外，我们展示了如何分析专家路由行为，揭示哪些MoE层能专门处理特定信息、哪些不能。这为保留关键层同时移除冗余提供了指导，从而在不牺牲精度的前提下提升ViMoE的效率。我们希望这项工作能为视觉MoE模型的设计提供新见解，并为未来研究提供有价值的实证指导。
英文摘要：
Mixture-of-Experts (MoE) models embody the divide-and-conquer concept and are a promising approach for increasing model capacity, demonstrating excellent scalability across multiple domains. In this paper, we integrate the MoE structure into the classic Vision Transformer (ViT), naming it ViMoE, and explore the potential of applying MoE to vision through a comprehensive study on image classification and semantic segmentation. However, we observe that the performance is sensitive to the configuration of MoE layers, making it challenging to obtain optimal results without careful design. The underlying cause is that inappropriate MoE layers lead to unreliable routing and hinder experts from effectively acquiring helpful information. To address this, we introduce a shared expert to learn and capture common knowledge, serving as an effective way to construct stable ViMoE. Furthermore, we demonstrate how to analyze expert routing behavior, revealing which MoE layers are capable of specializing in handling specific information and which are not. This provides guidance for retaining the critical layers while removing redundancies, thereby advancing ViMoE to be more efficient without sacrificing accuracy. We aspire for this work to offer new insights into the design of vision MoE models and provide valuable empirical guidance for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
