arXiv 2605.255652026-05-25cs.LGcs.CL

RotMoLE：通过旋转门控机制增强混合低秩专家

RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism

Tsinghua University（清华大学）；Beijing Information Science and Technology University（北京信息科技大学）；National University of Singapore（新加坡国立大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）

Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, Yihao Wang, Junpeng Liu, Yifan Zhu, Jie Tang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对MoE-LoRA中传统门控仅标量加权限制表示能力的问题，提出RotMoLE框架，通过引入旋转门控机制对每个专家进行旋转操作，提升专家利用率和专业化程度，在多任务和多语言训练中验证有效性。

摘要：

虽然大型语言模型（LLM）通常在进行垂直应用之前会针对特定领域任务进行微调，但将它们适应于具有多样化专业知识的复杂场景仍然具有挑战性。与此同时，混合专家（MoE）架构已成为训练LLM的关键范式，最近的一些工作也将MoE引入参数高效微调（PEFT），提出了混合低秩专家（MoE-LoRA），以增强低秩适配器学习复杂知识的能力。然而，MoE中的传统门控机制通常仅对选中的专家应用标量重新加权，从而限制了其表示和泛化的潜在能力。受MoE-LoRA中低秩结构的启发和推动，我们提出了RotMoLE，一个专门针对低秩专家的MoE框架，其特点是一个额外的旋转门控。除了简单的缩放，RotMoLE为每个选中的专家实现了一个旋转机制，从而在专家候选有限的情况下，实现了更好的专家利用和专业化，以学习多样化的数据。在复杂多任务和多语言训练场景下的实证结果验证了我们的有效性。

英文摘要：

While Large Language Models (LLMs) are commonly fine-tuned to handle domain-specific tasks before being applied to vertical applications, adapting them to complex scenarios with diverse specialized knowledge remains challenging. Meanwhile, Mixture-of-Experts (MoE) architecture has risen as a crucial paradigm for training LLMs, and some recent works have also incorporated MoE into Parameter-Efficient Fine-Tuning (PEFT) to propose the Mixture of Low-rank Experts (MoE-LoRA), to enhance the power of low-rank adapters for learning complicated knowledge. However, conventional gating mechanisms in MoE typically apply only a scalar reweighing to selected experts, thereby limiting their underlying capacity of representation and generalization. Motivated and enabled by the low-rank structures in MoE-LoRA, we propose RotMoLE, a specialized MoE framework for low-rank experts featuring an additional rotation gate. Beyond simple scaling, RotMoLE implements a rotation mechanism for each selected expert, enabling superior expert exploitation and specialization for learning diverse data, especially when expert candidates are limited. Empirical results on complex multi-task and multilingual training scenarios validate our effectiveness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。