K-Merge: 面向设备端大语言模型的在线持续适配器合并
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models
AI总结：
针对设备端大语言模型在线持续合并LoRA适配器的问题，提出一种无数据、计算高效的策略，在存储和计算受限下选择并合并新适配器，同时保持已有任务性能。
摘要：
设备端部署大语言模型通常利用低秩适配器在资源受限条件下支持多种下游任务。为解决移动设备存储容量有限的问题，近期工作探索了模型合并技术，将多个LoRA融合为一个。然而，实际中LoRA通常是增量交付的，因为用户会请求支持新任务（例如新问题类型或语言）。这种场景引入了一个新挑战：设备端在线持续合并，目标是在合并新LoRA的同时保持对已有任务的支持。本文提出一种无数据且计算高效的策略，用于在新适配器可用时选择并合并LoRA，假设设备只能存储有限数量的适配器。在真实任务上的大量实验表明，我们的方法在遵守设备端存储预算和计算限制的同时，优于其他策略。项目页面见：this https URL。
英文摘要：
On-device deployment of Large Language Models (LLMs) frequently leverages Low-Rank Adapters (LoRAs) to support diverse downstream tasks under tight resource constraints. To address the limited storage capacity of mobile devices, recent works have explored model merging techniques to fuse multiple LoRAs into a single one. In practice, however, LoRAs are often delivered incrementally, as users request support for new tasks (e.g., novel problem types or languages). This scenario introduces a new challenge: on-device online continual merging, where the objective is to incorporate new LoRAs while preserving the performance on previously supported tasks. In this paper, we propose a data-free and computationally efficient strategy for selecting and merging LoRAs when a new one becomes available, assuming the device can store only a limited number of adapters. Extensive experiments across real-world tasks demonstrate the superiority of our approach compared to alternative strategies while adhering to the storage budget and compute limitations of on-device settings. The project page is available at: https://donaldssh.github.io/K-Merge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
