arXiv 2510.178982026-01-07cs.LGcs.AI

L-MoE：轻量级低秩适应专家的端到端训练

L-MoE: End-to-End Training of a Lightweight Mixture of Low-Rank Adaptation Experts

Shihao Ji, Zihui Song

备注：: The authors have identified a technical error in the mathematical formulation regarding the differentiable routing mechanism described in Section 3.2. To ensure the accuracy and integrity of the scientific record, we wish to withdraw the paper to correct these formulations and re-evaluate the theoretical proofs

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

L-MoE通过端到端训练将低秩适应专家与混合专家架构结合，实现轻量级、高效且可扩展的语言模型训练方法。

摘要：

混合专家（MoE）架构通过为每个输入激活稀疏的权重子集，使大型语言模型（LLMs）能够扩展到万亿参数，同时保持推理过程中的计算成本恒定。同时，低秩适应（LoRA）已成为在专用任务上对LLMs进行参数高效微调的主导技术。在本工作中，我们统一了这两种范式，提出了一种名为L-MoE：轻量级LoRA专家的新型端到端可训练框架。L-MoE重新定义MoE专家，不再作为密集的前馈网络，而是作为一组任务专用的低秩适配器。一个轻量级门控网络，与专家共同训练，学习通过计算每个输入标记的加权平均参数动态组合这些LoRA适配器。这种组合是完全可微的，允许从标准自回归语言建模目标中回流梯度通过整个架构，同时优化专家适配器和路由策略。这种方法创建了一个高度参数高效的MoE模型，设计上具有模块性，允许动态技能组合，并且可以从端到端进行训练。我们提出了L-MoE的正式数学框架，详细说明了可微的路由机制和联合优化目标，从而为构建更高效、可扩展和专用的语言模型提供了新的路径。

英文摘要：

The Mixture of Experts (MoE) architecture enables the scaling of Large Language Models (LLMs) to trillions of parameters by activating a sparse subset of weights for each input, maintaining constant computational cost during inference. Concurrently, Low-Rank Adaptation (LoRA) has emerged as a dominant technique for parameter-efficiently fine-tuning LLMs on specialized tasks. In this work, we unify these two paradigms into a novel, end-to-end trainable framework named L-MoE: a Lightweight Mixture of LoRA Experts. L-MoE redefines MoE experts not as dense feed-forward networks, but as a collection of task-specialized, low-rank adapters. A lightweight gating network, trained jointly with the experts, learns to dynamically compose these LoRA adapters by computing a weighted average of their parameters for each input token. This composition is fully differentiable, allowing gradients from a standard auto-regressive language modeling objective to flow back through the entire architecture, simultaneously refining both the expert adapters and the routing strategy. This approach creates a highly parameter-efficient MoE model that is modular by design, allows for dynamic skill composition, and is trainable from end-to-end. We present the formal mathematical framework for L-MoE, detailing the differentiable routing mechanism and the joint optimization objective, thereby providing a new path toward building more efficient, scalable, and specialized language models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。