SLIM：通过Soft LoRA与Identity Mixture让LLM学得更多、忘得更少
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
AI总结：
提出基于Soft LoRA和Identity Mixture的MoE框架SLIM，通过动态路由、滑动聚类和快速动态合并，在保持下游性能的同时有效缓解灾难性遗忘。
摘要：
尽管已付出诸多努力，但在许多应用中平衡LLM的训练预算、下游性能和通用能力仍是一项挑战。针对下游任务训练整个模型代价高昂，且极易导致灾难性遗忘。通过引入参数高效微调（PEFT），虽能降低训练成本，但仍存在遗忘问题，且限制了对下游任务的学习。为了在微调LLM时减少对下游性能的限制并缓解通用能力的遗忘，我们提出了一种基于Soft LoRA和Identity Mixture（SLIM）的新型混合专家框架，该框架允许在LoRA适配器和跳跃连接之间进行动态路由，从而实现对遗忘的抑制。我们采用带有滑动聚类的权重生成机制，以获得更好的域外区分能力来增强路由。我们还提出将低秩适配器的混合转换为模型合并公式，并引入LoRA适配器的快速动态合并，以保持基础模型的通用能力。大量实验表明，所提出的SLIM在下游任务上与最先进的PEFT方法相当，同时在缓解灾难性遗忘方面取得了领先性能。
英文摘要：
Although many efforts have been made, it is still a challenge to balance the training budget, downstream performance, and the general capabilities of the LLMs in many applications. Training the whole model for downstream tasks is expensive, and could easily result in catastrophic forgetting. By introducing parameter-efficient fine-tuning (PEFT), the training cost could be reduced, but it still suffers from forgetting, and limits the learning on the downstream tasks. To efficiently fine-tune the LLMs with less limitation to their downstream performance while mitigating the forgetting of general capabilities, we propose a novel mixture of expert (MoE) framework based on Soft LoRA and Identity Mixture (SLIM), that allows dynamic routing between LoRA adapters and skipping connection, enables the suppression of forgetting. We adopt weight-yielding with sliding clustering for better out-of-domain distinguish to enhance the routing. We also propose to convert the mixture of low-rank adapters to the model merging formulation and introduce fast dynamic merging of LoRA adapters to keep the general capabilities of the base model. Extensive experiments demonstrate that the proposed SLIM is comparable to the state-of-the-art PEFT approaches on the downstream tasks while achieving the leading performance in mitigating catastrophic forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
