arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.02656cs.CVcs.LG
收藏

GTA:从以对象为中心的表征中引导空间注意力的迁移

GTA: Guided Transfer of Spatial Attention from Object-Centric Representations

LG CNS AI Research(LG CNS AI研究院);Seoul National University of Science and Technology(首尔科学技术大学)

SeokHyun Seo, Jinwoo Hong, JungWoo Chae, Kyungyul Kim, Sangheum Hwang

AI总结:

针对ViT在小数据集上易过拟合导致迁移表征退化的问题,提出空间注意力引导迁移(GTA)方法,通过正则化源模型与目标模型间的自注意力图,使目标模型有效利用对象定位知识,在五个基准数据集上显著提升了准确率。

摘要:

在迁移学习中利用训练良好的表征,通常比从头开始训练能带来更优的性能和更快的收敛速度。然而,即使迁移了这些良好的表征,模型也很容易在有限的训练数据集上过拟合,并失去迁移表征的宝贵特性。由于ViT的归纳偏置较低,这一现象在ViT中更为严重。通过在ViT中使用注意力图进行实验分析,我们观察到,在小数据集上训练时,丰富的表征会发生退化。受此发现的启发,我们提出了一种新颖且简单的ViT正则化方法,称为空间注意力引导迁移(GTA)。我们提出的方法对源模型和目标模型之间的自注意力图进行正则化。通过这种显式正则化,目标模型能够充分利用与对象定位特性相关的知识。我们的实验结果表明,所提出的GTA在五个基准数据集上持续提高了准确率,尤其是在训练数据量较少的情况下。

英文摘要:

Utilizing well-trained representations in transfer learning often results in superior performance and faster convergence compared to training from scratch. However, even if such good representations are transferred, a model can easily overfit the limited training dataset and lose the valuable properties of the transferred representations. This phenomenon is more severe in ViT due to its low inductive bias. Through experimental analysis using attention maps in ViT, we observe that the rich representations deteriorate when trained on a small dataset. Motivated by this finding, we propose a novel and simple regularization method for ViT called Guided Transfer of spatial Attention (GTA). Our proposed method regularizes the self-attention maps between the source and target models. A target model can fully exploit the knowledge related to object localization properties through this explicit regularization. Our experimental results show that the proposed GTA consistently improves the accuracy across five benchmark datasets especially when the number of training data is small.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑