参数高效路由微调：混合专家模型需要混合适配模块
Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
AI总结：
针对现有PEFT未利用MoE动态路由的问题，提出在适配模块中融入路由机制，并在OLMoE-1B-7B和Mixtral-8x7B上验证了其性能与效率，给出了最优配置与实用见解。
摘要：
混合专家（MoE）模型得益于其专家间的动态路由机制，而现有的参数高效微调（PEFT）策略未能利用这一机制。这促使我们研究适配模块本身是否应融入路由机制，以与MoE的多专家架构对齐。我们分析了将PEFT应用于MoE语言模型时核心组件的动态变化，并考察了不同路由策略对适配效果的影响。通过在OLMoE-1B-7B和Mixtral-8x7B上针对多种常识与数学推理任务进行的大量实验，验证了我们路由方法的性能与效率。我们为不同场景确定了最优配置，并提供了实证分析与实用见解，以促进更好的PEFT和MoE应用。
英文摘要：
Mixture-of-Experts (MoE) benefits from a dynamic routing mechanism among their specialized experts, which existing Parameter- Efficient Fine-Tuning (PEFT) strategies fail to leverage. This motivates us to investigate whether adaptation modules themselves should incorporate routing mechanisms to align with MoE's multi-expert architecture. We analyze dynamics of core components when applying PEFT to MoE language models and examine how different routing strategies affect adaptation effectiveness. Extensive experiments adapting OLMoE-1B-7B and Mixtral-8x7B on various commonsense and math reasoning tasks validate the performance and efficiency of our routed approach. We identify the optimal configurations for different scenarios and provide empirical analyses with practical insights to facilitate better PEFT and MoE applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
