MoLoRA:通过每token适配器路由实现可组合的专精
MoLoRA: Composable Specialization via Per-Token Adapter Routing
AI总结:
MoLoRA通过每token适配器路由实现可组合专精,优于规模扩展:MoLoRA使Qwen3-1.7B在四个推理基准上超越Qwen3-8B,且小4.7倍。
摘要:
多适配器服务系统将整个序列路由到单一适配器,迫使在请求跨多个领域时做出选择。这一假设在两个重要场景中失效:(1)多模态生成,其中文本和图像token需要不同的适配器在同一序列中;(2)混合能力请求如“编写代码解决此方程”,需要多个专精适配器的专家知识。我们引入了每token路由,根据词汇结构(用于多模态模型)或学习的门控(用于语义专精)将单个token路由到适配器。每token路由被证明是最优的,实现N tokens for N work versus K·N for per-sequence routing with K adapter types。我们的关键贡献是MoLoRA(LoRA的混合),它实现了可组合的专精:加载多个领域专用适配器,并让学习的路由器按token选择合适的适配器。我们证明专精显著优于规模扩展:MoLoRA使Qwen3-1.7B在四个推理基准上超越Qwen3-8B,同时小4.7倍。这使推理时实现模块化专精:独立训练专注LoRAs,无需重新训练即可组合它们,并通过简单加载新适配器添加新能力。
英文摘要:
Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。