利用低秩适配器混合探索异构数据训练
Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
AI总结：
本文提出低秩适配器混合（MoLA）方法，通过 MoLA-Grad 和 MoLA-Router 两种变体分别处理目标已知与未知场景，利用任务标识符和任务间去相关损失缓解异构数据训练冲突，实验验证其优于现有方法。
摘要：
训练一个统一模型以同时考虑多个目标，是迈向通用人工智能的一种趋势。然而，如何高效缓解来自不同领域或任务所收集的异构数据之间的训练冲突，仍未得到充分探索。在本研究中，我们探索利用低秩适配器混合（MoLA）来缓解异构数据训练中的冲突，这需要联合训练多个低秩适配器及其共享骨干网络。具体而言，我们引入 MoLA 的两种变体，即 MoLA-Grad 和 MoLA-Router，分别处理推理阶段的目标已知和目标未知场景。前者使用任务标识符为每个任务分配个性化低秩适配器，将任务特定知识解耦到各自的适配器中，从而缓解异构性冲突。后者使用一种新颖的任务间去相关（TwD）损失来干预路由器，使其学习面向同质任务的适配器定向权重组合，达到类似效果。我们进行了全面的实验，验证了 MoLA 相对于先前最先进方法的优越性，并对其工作机制进行了深入分析。源代码可在以下地址获取：https://github.com/MediaBrain-SJTU/MoLA
英文摘要：
Training a unified model to take multiple targets into account is a trend towards artificial general intelligence. However, how to efficiently mitigate the training conflicts among heterogeneous data collected from different domains or tasks remains under-explored. In this study, we explore to leverage Mixture of Low-rank Adapters (MoLA) to mitigate conflicts in heterogeneous data training, which requires to jointly train the multiple low-rank adapters and their shared backbone. Specifically, we introduce two variants of MoLA, namely, MoLA-Grad and MoLA-Router, to respectively handle the target-aware and target-agnostic scenarios during inference. The former uses task identifiers to assign personalized low-rank adapters to each task, disentangling task-specific knowledge towards their adapters, thereby mitigating heterogeneity conflicts. The latter uses a novel Task-wise Decorrelation (TwD) loss to intervene the router to learn oriented weight combinations of adapters to homogeneous tasks, achieving similar effects. We conduct comprehensive experiments to verify the superiority of MoLA over previous state-of-the-art methods and present in-depth analysis on its working mechanism. Source code is available at: https://github.com/MediaBrain-SJTU/MoLA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
