增量低秩更新下预训练模型遗忘现象的实证分析
An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates
AI总结：
该研究聚焦预训练模型经LoRA合并权重的增量更新场景，实证分析LoRA秩对预训练及下游任务遗忘的影响，发现视觉Transformer存在残差网络未表现出的情境式遗忘新现象。
摘要：
通过HuggingFace等平台，大型预训练基础模型在互联网上广泛开源可用，这一趋势已席卷实用深度学习领域。如今，神经网络训练的经典流程通常是在小型目标数据集上微调这些预训练网络，而非从头开始训练。对于大模型，使用被称为低秩适配（Low-Rank Adaptation, LoRA）的低秩训练技术，即便在普通硬件上也能完成微调。尽管低秩训练已在持续学习场景中被研究，但现有工作往往考虑将学习到的适配器与现有模型一同存储，却很少尝试在每个任务训练完成后，通过将LoRA与现有权重合并来修改预训练模型的权重。本文研究了这一场景，探讨了LoRA秩对预训练基础任务的遗忘、以及对后续任务的可塑性和遗忘的影响。我们观察到，该秩对预训练任务和下游任务的遗忘都有重要影响。我们还发现，以这种方式微调的视觉Transformer表现出一种“情境式”遗忘行为，这种行为在残差网络中并未观察到，且我们认为在以往的持续学习研究中尚未被发现。
英文摘要：
Broad, open source availability of large pretrained foundation models on the internet through platforms such as HuggingFace has taken the world of practical deep learning by storm. A classical pipeline for neural network training now typically consists of finetuning these pretrained network on a small target dataset instead of training from scratch. In the case of large models this can be done even on modest hardware using a low rank training technique known as Low-Rank Adaptation (LoRA). While Low Rank training has already been studied in the continual learning setting, existing works often consider storing the learned adapter along with the existing model but rarely attempt to modify the weights of the pretrained model by merging the LoRA with the existing weights after finishing the training of each task. In this article we investigate this setting and study the impact of LoRA rank on the forgetting of the pretraining foundation task and on the plasticity and forgetting of subsequent ones. We observe that this rank has an important impact on forgetting of both the pretraining and downstream tasks. We also observe that vision transformers finetuned in that way exhibit a sort of ``contextual'' forgetting, a behaviour that we do not observe for residual networks and that we believe has not been observed yet in previous continual learning works.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
