arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.21795cs.LGcs.AI
收藏

利用任务表示的有效LoRA适配器路由

Effective LoRA Adapter Routing using Task Representations

EPFL(苏黎世联邦理工学院)

Akash Dhasade, Anne-Marie Kermarrec, Igor Pavlovic, Diana Petrescu, Rafael Pires, Mathis Randl, Martijn de Vos

AI总结:

LORAUTER通过任务表示实现高效适配器路由,优于现有方法并在未见过的任务上取得最佳成绩。

摘要:

低秩适应(LoRA)通过模块化适配器实现了大型语言模型(LLMs)的参数高效专业化,导致公共适配器池迅速增长,覆盖了多样化的任务。有效使用这些适配器需要路由:选择和组合适当的适配器以处理查询。我们引入了LORAUTER,一种新的路由框架,通过任务表示选择和组合LoRA适配器,而不是适配器特性。与现有方法直接将查询映射到适配器不同,LORAUTER通过从小型验证集中衍生的任务嵌入来路由查询,并且不需要适配器训练数据。通过在任务层面操作,LORAUTER实现了高效的路由,其扩展性与任务数量而非适配器数量成正比。在多个任务上的实验表明,LORAUTER在基准路由方法上表现一致,当存在任务对齐的适配器时,其性能可达到Oracle水平(101.2%),并在未见过的任务上实现了最先进的结果(+5.2分)。我们进一步展示了LORAUTER在非常大的、嘈杂的适配器池中鲁棒性的能力,通过将其扩展到超过1500个适配器。

英文摘要:

Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑