JumpLoRA: 大型语言模型中持续学习的稀疏适配器
JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
AI总结：
JumpLoRA通过JumpReLU门控在LoRA块中诱导稀疏性，实现动态参数隔离，提升IncLoRA性能并超越ELLA方法。
摘要：
基于适配器的方法已成为大语言模型（LLMs）持续学习（CL）的有效途径，通过为每个任务顺序学习低秩更新矩阵。为缓解灾难性遗忘，最新方法通过子空间或坐标干扰方面对新适配器施加约束。本文提出JumpLoRA，一种通过JumpReLU门控在LoRA块中自适应诱导稀疏性的新框架。该方法实现了动态参数隔离，有助于防止任务干扰。我们证明了该方法高度模块化且兼容LoRA-based CL方法。具体而言，它显著提升了IncLoRA的性能，并超越了最先进的持续学习方法ELLA。
英文摘要：
Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
