arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.05454cs.CVcs.AI
收藏

视觉变换器中的注意力保留用于持续学习

Attention Retention for Continual Learning with Vision Transformers

Yue Lu, Xiangyu Zhou, Shizhou Zhang, Yinghui Xing, Guoqiang Liang, Wencong Zhang

AI总结:

本文提出一种基于视觉变换器的注意力保留框架,通过约束注意力漂移来减轻持续学习中的灾难性遗忘,通过生成实例自适应二进制掩码并调整梯度更新,有效保留已学习的视觉概念。

摘要:

持续学习(CL)使人工智能系统能够从非平稳数据流中逐步获取知识。然而,灾难性遗忘仍然是一个关键挑战。在本工作中,我们识别出视觉变换器中的注意力漂移是灾难性遗忘的主要来源,其中在学习新任务后,对之前学习的视觉概念的注意力会显著变化。受神经科学对人类视觉系统选择性注意的见解启发,我们提出了一种新的注意力保留框架,以减轻持续学习中的遗忘。我们的方法通过在反向传播过程中显式修改梯度来约束注意力漂移,通过两个步骤:1)使用逐层展开机制提取先前任务的注意力图并生成实例自适应的二进制掩码;2)在学习新任务时,应用这些掩码以零出与先前注意力区域相关的梯度,从而防止已学习的视觉概念受到干扰。为了与现代优化器兼容,梯度掩码过程进一步通过按比例缩放参数更新来增强,以保持其相对大小。实验和可视化显示了我们的方法在减轻灾难性遗忘和保留视觉概念方面的有效性。它在持续学习任务中实现了最先进的性能,并在多种持续学习场景中表现出稳健的泛化能力。

英文摘要:

Continual learning (CL) empowers AI systems to progressively acquire knowledge from non-stationary data streams. However, catastrophic forgetting remains a critical challenge. In this work, we identify attention drift in Vision Transformers as a primary source of catastrophic forgetting, where the attention to previously learned visual concepts shifts significantly after learning new tasks. Inspired by neuroscientific insights into the selective attention in the human visual system, we propose a novel attention-retaining framework to mitigate forgetting in CL. Our method constrains attention drift by explicitly modifying gradients during backpropagation through a two-step process: 1) extracting attention maps of the previous task using a layer-wise rollout mechanism and generating instance-adaptive binary masks, and 2) when learning a new task, applying these masks to zero out gradients associated with previous attention regions, thereby preventing disruption of learned visual concepts. For compatibility with modern optimizers, the gradient masking process is further enhanced by scaling parameter updates proportionally to maintain their relative magnitudes. Experiments and visualizations demonstrate the effectiveness of our method in mitigating catastrophic forgetting and preserving visual concepts. It achieves state-of-the-art performance and exhibits robust generalizability across diverse CL scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑