LoRA-Mixer：通过序列注意力路由协调模块化LoRA专家
LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
AI总结：
LoRA-Mixer通过将任务特定的LoRA专家路由到注意力模块的核心投影矩阵中，实现细粒度的token级专业化，同时保持与Transformer和状态空间模型的兼容性，且在15个基准测试中表现优于现有方法。
摘要：
近期尝试将低秩适应（LoRA）与混合专家（MoE）结合以实现大语言模型（LLMs）的多任务适应时，常将整个注意力/FFN层替换为切换专家或添加并行专家分支，这会削弱参数效率并限制任务专业化。我们引入LoRA-Mixer，一种模块化MoE框架，将任务特定的LoRA专家路由到注意力模块的核心投影矩阵，即输入和输出线性层，而非主要针对FFN块。该设计通过充分利用注意力机制实现细粒度的token级专业化，同时保持与Transformer和状态空间模型（SSMs）的滴入兼容性，因为线性投影层普遍存在。为了从有限的数据中训练鲁棒的路由器，同时促进稳定、选择性的决策和高专家复用，LoRA-Mixer采用了一种自适应的路由专业化损失（RSL），通过熵塑造目标联合强制全局负载平衡和输入感知专业化。该框架支持两种模式：（i）联合优化适配器和路由器，使用可微的硬-软top-k路由方案；（ii）在冻结的预训练LoRA模块上进行即插即用的路由，这些模块来自公共仓库。在15个基准测试中，包括MedQA、GSM8K、HumanEval和GLUE，RSL优化的LoRA-Mixer在使用其可训练参数的48%时，优于现有路由和LoRA-MoE基线，分别在GSM8K、CoLA和ARC-C上获得3.79、2.90和3.95个百分点的提升。跨模型迁移和适配器复用实验进一步证明了该方法的通用性和数据效率。我们的代码可在https://github.com/hustcselwb/LoRA-Mixer获取。
英文摘要：
Recent attempts to combine low-rank adaptation (LoRA) with mixture-of-experts (MoE) for multi-task adaptation of Large Language Models (LLMs) often replace whole attention/FFN layers with switch experts or append parallel expert branches, undermining parameter efficiency and limiting task specialization. We introduce LoRA-Mixer, a modular MoE framework that routes task-specific LoRA experts into the core projection matrices of the attention module, namely input and output linear layers, rather than primarily targeting FFN blocks. The design delivers fine-grained token-level specialization by fully exploiting the attention mechanism, while remaining drop-in compatible with Transformers and state-space models (SSMs), since linear projection layers are ubiquitous. To train robust routers from limited data while promoting stable, selective decisions and high expert reuse, LoRA-Mixer employs an adaptive Routing Specialization Loss (RSL) that jointly enforces global load balance and input-aware specialization via an entropy-shaping objective. The framework supports two regimes: (i) joint optimization of adapters and router with a differentiable hard-soft top-k routing scheme, and (ii) plug-and-play routing over frozen, pre-trained LoRA modules sourced from public repositories. Across 15 benchmarks, including MedQA, GSM8K, HumanEval, and GLUE, RSL-optimized LoRA-Mixer outperforms state-of-the-art routing and LoRA-MoE baselines while using 48 percent of their trainable parameters, with gains of 3.79, 2.90, and 3.95 percentage points on GSM8K, CoLA, and ARC-C, respectively. Cross-model transfer and adapter reuse experiments further demonstrate the approach's versatility and data efficiency. Our code is available at https://github.com/hustcselwb/LoRA-Mixer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
