arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.08457cs.CVcs.AIcs.LG
收藏

用均匀注意力为视觉Transformer“挠背”

Scratching Visual Transformer's Back with Uniform Attention

POSTECH(浦项科技大学);NAVER AI Lab(NAVER AI实验室);Tübingen University(蒂宾根大学);Yonsei University(延世大学)

Nam Hyeon-Woo, Kim Yu-Ji, Byeongho Heo, Dongyoon Han, Seong Joon Oh, Tae-Hyun Oh

AI总结:

本文发现ViT偏好密集注意力交互,提出在各层插入均匀注意力的上下文广播(CB)方法,以极低成本提升模型容量和泛化能力。

摘要:

视觉Transformer(ViTs)的优异性能通常归因于多头自注意力(MSA)。MSA在ViT模型的每一层实现全局交互,这与卷积神经网络(CNN)跨多层逐渐扩大交互范围的特征形成对比。我们研究了注意力密度的作用。初步分析表明,注意力图的空间交互接近密集交互而非稀疏交互。这是一个有趣的现象,因为密集注意力图由于softmax梯度在其周围更陡峭而更难让模型学习。我们将此解释为ViT模型对包含密集交互的强烈偏好。因此,我们手动将均匀注意力插入ViT模型的每一层,以提供急需的密集交互。我们将此方法称为上下文广播(Context Broadcasting,CB)。我们观察到,加入CB降低了原始注意力图中的密度程度,并同时提高了ViT模型的容量和泛化能力。CB带来的成本可忽略不计:模型代码中仅需1行,无需额外参数,且额外操作极少。

英文摘要:

The favorable performance of Vision Transformers (ViTs) is often attributed to the multi-head self-attention (MSA). The MSA enables global interactions at each layer of a ViT model, which is a contrasting feature against Convolutional Neural Networks (CNNs) that gradually increase the range of interaction across multiple layers. We study the role of the density of the attention. Our preliminary analyses suggest that the spatial interactions of attention maps are close to dense interactions rather than sparse ones. This is a curious phenomenon, as dense attention maps are harder for the model to learn due to steeper softmax gradients around them. We interpret this as a strong preference for ViT models to include dense interaction. We thus manually insert the uniform attention to each layer of ViT models to supply the much needed dense interactions. We call this method Context Broadcasting, CB. We observe that the inclusion of CB reduces the degree of density in the original attention maps and increases both the capacity and generalizability of the ViT models. CB incurs negligible costs: 1 line in your model code, no additional parameters, and minimal extra operations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑