arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.02509cs.AI
收藏

FCL-ViT:面向持续学习的任务感知注意力调优

FCL-ViT: Task-Aware Attention Tuning for Continual Learning

Aristotle University of Thessaloniki(亚里士多德大学)

Anestis Kaimakamidis, Ioannis Pitas

AI总结:

本文提出FCL-ViT,通过反馈机制与可调自注意力块实现任务感知的动态注意力调优,在持续学习中超越现有方法并保持较少可训练参数。

摘要:

持续学习(CL)涉及将先前的深度神经网络(DNN)知识适配到新任务,同时不遗忘旧任务。然而,现代持续学习技术侧重于为现有DNN模型提供记忆能力,而不是设计能够根据当前任务进行适配的新模型。本文提出了新颖的反馈持续学习视觉Transformer(FCL-ViT),它利用反馈机制生成针对当前任务定制的实时动态注意力特征。FCL-ViT分两个阶段运行。在阶段1中,生成通用图像特征,并确定Transformer应在当前图像上关注的位置。在阶段2中,生成利用动态注意力的任务特定图像特征。为此,引入了可调自注意力块(TABs)和任务特定块(TSBs),它们分别在两个阶段中运行,并负责调优TABs的注意力。与基准方法相比,FCL-ViT在持续学习上超越了最先进的性能,同时保留了少量的可训练DNN参数。

英文摘要:

Continual Learning (CL) involves adapting the prior Deep Neural Network (DNN) knowledge to new tasks, without forgetting the old ones. However, modern CL techniques focus on provisioning memory capabilities to existing DNN models rather than designing new ones that are able to adapt according to the task at hand. This paper presents the novel Feedback Continual Learning Vision Transformer (FCL-ViT) that uses a feedback mechanism to generate real-time dynamic attention features tailored to the current task. The FCL-ViT operates in two Phases. In phase 1, the generic image features are produced and determine where the Transformer should attend on the current image. In phase 2, task-specific image features are generated that leverage dynamic attention. To this end, Tunable self-Attention Blocks (TABs) and Task Specific Blocks (TSBs) are introduced that operate in both phases and are responsible for tuning the TABs attention, respectively. The FCL-ViT surpasses state-of-the-art performance on Continual Learning compared to benchmark methods, while retaining a small number of trainable DNN parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑