发表机构
Samsung Research America（三星美国研究院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对LoRA合并中均匀秩预算假设导致的性能损失，提出无数据净效用指标，通过SVD分解和全局评分选择奇异方向，在视觉和语言任务上分别提升2.1%和2.2%。
AI 中文摘要
合并低秩适配器（LoRA）有望消除推理时交换任务特定权重的开销。然而，现有的合并方法假设每一层需要相同的秩预算。此外，一些方法还假设秩预算需要在任务之间平均分配。我们证明这种均匀预算假设是合并后的LoRA与每任务LoRA之间性能差距的主要来源。然而，秩选择是一个NP难问题。为此，我们引入了净效用（Net Utility），一种无数据指标，首先通过奇异值分解（SVD）对每个任务LoRA进行分解，并根据其任务效用及其与其他任务方向的干扰对每个奇异方向进行评分。接下来，我们在全局范围内汇集这些分数，在所选方向总数受限的条件下选择具有最高值的奇异方向。所提出的净效用指标应用于跨越三个不同合并空间的五种不同合并方法之上。合并是在两组任务（视觉任务和语言任务）上进行的。基于净效用的秩分配优于没有该分配的对应方法。平均而言，在视觉任务上实现了+2.1%的性能提升，在语言任务上实现了+2.2%的提升。
英文摘要
Merging low-rank adapters (LoRAs) promises to eliminate the overhead of swapping task-specific weights at inference time. However, existing merging methods assume every layer needs the same rank budget. Further, some methods assume that rank budget needs to be split equally among the tasks too. We show this uniform-budget assumption is a major source of the performance gap between merged and per-task LoRAs. However, rank selection is an NP hard problem. To this end, we introduce Net Utility, a data free metric that first decomposes every task LoRA by its Singular Value Decomposition (SVD) and scores each of those singular directions by its task utility and its interference with other tasks directions. Next, we globally pool these scores to select singular directions with the highest values with a constraint on the total number of directions selected. The proposed Net Utility metric is applied on top of five different merging methods across three different merging spaces. The merging is done over two sets of tasks, vision and language tasks. Net utility based rank allocation outperforms its counterparts without that allocation. On average, over vision tasks it achieves +2.1% improvement in performance, and +2.2% improvement over the language tasks.
Comments7 pages, AAAI submission
