最少但不最后：细调中间主成分以获得更好的性能-遗忘权衡
Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs
AI总结：
本文提出了一种基于中间主成分的LoRA初始化方法，通过细调中间组件实现更好的性能-遗忘权衡，提升了模型的准确性和鲁棒性。
摘要：
低秩适应（LoRA）方法已成为在计算和内存限制下适应大型预训练模型以适应下游任务的关键技术。然而，它们面临一个根本性的挑战，即在任务特定性能提升与预训练知识的灾难性遗忘之间取得平衡，而现有方法提供不一致的建议。本文全面分析了低秩适应中固有的性能-遗忘权衡，使用主成分作为初始化。我们的研究发现，细调中间组件在平衡和对高学习率的鲁棒性方面优于现有工作的第一（PiSSA）和最后（MiLoRA）组件。基于这些发现，我们提供了一种实用的LoRA初始化方法，提供了更优的权衡。我们在各种计算机视觉和NLP任务上的彻底实证研究中证明，我们的方法提高了准确性并减少了遗忘，也适用于持续学习场景。
英文摘要：
Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in balancing task-specific performance gains against catastrophic forgetting of pre-trained knowledge, where existing methods provide inconsistent recommendations. This paper presents a comprehensive analysis of the performance-forgetting trade-offs inherent in low-rank adaptation using principal components of weight matrices as initialization. Our investigation reveals that fine-tuning intermediate components leads to better balance and robustness to high learning rates than first (PiSSA) and last (MiLoRA) components in existing work. Building on these findings, we provide practical guidelines for initialization of LoRA methods to balance the performance-forgetting trade-off. In a thorough empirical study on a variety of computer vision and NLP tasks we confirm that these guidelines achieve high accuracy and reduced forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
