arXiv 2603.096842026-03-10cs.LG

关于基于低秩分解的参数高效微调中的灾难性遗忘

On Catastrophic Forgetting in Low-Rank Decomposition-Based Parameter-Efficient Fine-Tuning

Department of Electrical and Computer Engineering, The University of British Columbia（英国不列颠哥伦比亚大学电气与计算机工程系）；Department of Mathematics, The University of British Columbia（英国不列颠哥伦比亚大学数学系）；Department of Chemical and Biological Engineering, The University of British Columbia（英国不列颠哥伦比亚大学化学与生物工程系）

Muhammad Ahmad, Jingjing Zheng, Yankai Cao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究揭示了基于低秩分解的参数高效微调中，更新子空间设计对减少灾难性遗忘的关键作用，并提出结构对齐参数化方法以提升持续学习效果。

摘要：

基于低秩分解的参数高效微调（PEFT）技术，如LoRA，已成为适应大型预训练模型的标准方法。然而，其在序列学习中的行为，特别是关于灾难性遗忘方面，仍然不够了解。在本工作中，我们通过实证研究显示，遗忘程度强烈受到更新子空间的几何结构和参数化的影响。尽管限制更新到小共享矩阵子空间的方法常遭受任务干扰，基于张量的分解（如LoRETTA）通过在超紧凑预算内捕捉更丰富的结构信息来减轻遗忘，而结构对齐的参数化（如WeGeFT）则保留预训练表示。我们的发现突显了更新子空间设计在持续学习中的关键作用，并为在序列设置中选择高效的适应策略提供了实用指导。

英文摘要：

Parameter-efficient fine-tuning (PEFT) based on low-rank decomposition, such as LoRA, has become a standard for adapting large pretrained models. However, its behavior in sequential learning -- specifically regarding catastrophic forgetting -- remains insufficiently understood. In this work, we present an empirical study showing that forgetting is strongly influenced by the geometry and parameterization of the update subspace. While methods that restrict updates to small, shared matrix subspaces often suffer from task interference, tensor-based decompositions (e.g., LoRETTA) mitigate forgetting by capturing richer structural information within ultra-compact budgets, and structurally aligned parameterizations (e.g., WeGeFT) preserve pretrained representations. Our findings highlight update subspace design as a key factor in continual learning and offer practical guidance for selecting efficient adaptation strategies in sequential settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。