CURLoRA：稳定的LLM持续微调与灾难性遗忘缓解
CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
AI总结：
本文提出CURLoRA方法，通过在LoRA中引入改进的CUR矩阵分解作为隐式正则化并仅微调零初始化的$U$矩阵，有效缓解了LLM持续微调中的灾难性遗忘，显著减少了可训练参数，并在数据有限时保持模型稳定性和准确率。
摘要：
本文介绍了CURLoRA，这是一种微调大型语言模型（LLMs）的新方法，它在低秩适应（LoRA）的背景下利用了CUR矩阵分解。我们的方法解决了LLM微调中的两个关键挑战：减轻持续学习期间的灾难性遗忘以及减少可训练参数的数量。我们对CUR分解过程提出了一种独特的修改，利用逆概率进行列和行的选择，这起到了隐式正则化的作用，并将$U$矩阵初始化为零矩阵，且仅对其进行微调。我们通过在多个数据集上的实验证明，CURLoRA在缓解灾难性遗忘方面优于标准LoRA。它在保持跨任务的模型稳定性和性能的同时，显著减少了可训练参数的数量。我们的结果表明，与LoRA在持续微调上相比，CURLoRA实现了非常好且稳定的任务准确率，同时保持基础模型的困惑度分数固定，特别是在数据有限的场景中。
英文摘要：
This paper introduces CURLoRA, a novel approach to fine-tuning large language models (LLMs) that leverages CUR matrix decomposition in the context of Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM fine-tuning: mitigating catastrophic forgetting during continual learning and reducing the number of trainable parameters. We propose a unique modification to the CUR decomposition process, utilizing inverted probabilities for column and row selection which acts as an implicit regularization, and initializing the $U$ matrix as a zero matrix, and only fine-tuning it. We demonstrate through experiments on multiple datasets that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting. It maintains model stability and performance across tasks while significantly reducing the number of trainable parameters. Our results show that CURLoRA achieves very good and stable task accuracy while maintaining base model's perplexity scores fixed compared to LoRA upon continual fine-tuning, particularly in scenarios with limited data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
