路由还不够：诊断MoE+LoRA微调中的适配器内子空间竞争
Routing Is Not Enough: Diagnosing Intra-Adapter Subspace Contention in MoE+LoRA Fine-Tuning
- Islamic University of Technology（伊斯兰理工大学）
- York University（约克大学）
- Dialpad Inc.（Dialpad公司）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究针对MoE+LoRA微调中路由分离无法防止负迁移的问题，提出SpawnLoRA方法，通过在MoE专家内动态添加门控子适配器减少负迁移，在Phi-tiny-MoE-instruct等模型上验证了其有效性。
AI中文摘要：
多领域微调常结合MoE路由与LoRA，假设token级路由可分离特定领域的更新。我们使用Python代码、生物医学文本和数学推理在MoE+LoRA中验证该假设，尽管这些领域呈现近乎不相交的专家路由，但添加生物医学数据会显著提升代码困惑度，表明仅路由分离可能无法防止负迁移。为定位失败原因，我们引入Jaccard路由重叠度和适配器梯度余弦相似度，分别衡量专家共享度和更新兼容性。这些诊断显示，干扰主要源于几乎正交的领域梯度在同一低秩适配器子空间内竞争。我们通过SpawnLoRA解决该问题，当检测到适配器级竞争时，它会在MoE专家内部动态添加门控子适配器，同时保持路由固定。我们在Phi-tiny-MoE-instruct和OLMoE-1B-7E上跨多个混合设置评估SpawnLoRA，发现与标准LoRA和秩自适应LoRA相比，它能有效减少负迁移。这些结果表明，专家内部的结构分离带来的益处超出了单独路由或秩扩展的范畴。
英文摘要：
Multi-domain fine-tuning often combines MoE routing with LoRA, assuming that token-level routing separates domain-specific updates. We test this assumption in MoE+LoRA using Python code paired with biomedical text and mathematical reasoning. Although these domains show near-disjoint expert routing, adding biomedical data substantially increases code perplexity, indicating that routing separation alone may not prevent negative transfer. To localize the failure, we introduce Jaccard routing overlap and adapter-gradient cosine similarity, which measure expert sharing and update compatibility, respectively. These diagnostics indicate that interference arises mostly from nearly orthogonal domain gradients competing within the same low-rank adapter subspace. We address this issue with SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts when adapter-level contention is detected, while keeping the router fixed. We evaluate SpawnLoRA on Phi-tiny-MoE-instruct and OLMoE-1B-7B across multiple mixture settings and find that it effectively reduces negative transfer compared with standard and rank-adaptive LoRA. These results demonstrate that structural separation inside experts provides benefits beyond routing or rank expansion alone.
