LoRI：减少多任务低秩适配中的跨任务干扰
LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
AI总结：
LoRI通过冻结随机投影矩阵A和稀疏化矩阵B，减少多任务LoRA的跨任务干扰，降低95%可训练参数，在多项任务上超越全量微调和现有PEFT方法。
摘要：
低秩适配（LoRA）已成为大型语言模型（LLMs）中一种流行的参数高效微调（PEFT）方法，然而它在多任务场景下仍会产生显著的额外开销并遭受参数干扰。我们提出了具有减少干扰的LoRA（LoRI），这是一种简单而有效的方法，它将投影矩阵$A$冻结为随机投影，并使用任务特定的掩码对矩阵$B$进行稀疏化。这种设计大幅减少了可训练参数的数量，同时保持了强大的任务性能。此外，LoRI通过利用适配器子空间之间的正交性，最小化了适配器合并中的跨任务干扰，并通过使用稀疏性来缓解灾难性遗忘，支持持续学习。在自然语言理解、数学推理、代码生成和安全对齐任务上的大量实验表明，LoRI优于全量微调和现有的PEFT方法，同时使用的可训练参数比LoRA少高达95%。在多任务实验中，LoRI实现了有效的适配器合并和持续学习，并减少了跨任务干扰。代码可在以下网址获取：https://github.com/juzhengz/LoRI
英文摘要：
Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
