arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.12731cs.CV
收藏

用于参数有效视觉骨干的缩放局部自注意

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Google Research(谷歌研究院);UC Berkeley(加州大学伯克利分校)

Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, Jonathon Shlens

摘要:

与卷积的参数依赖性缩放和内容独立性交互相比,自注意具有改善计算机视觉系统的前景,这是由于感受野的参数独立缩放和内容依赖性交互。最近的研究表明,与ResNet-50等基线卷积模型相比,自注意力模型在准确性参数权衡方面有了令人鼓舞的改进。在这项工作中,我们的目标是开发自我注意力模型,不仅可以超越规范的基线模型,甚至可以超越高性能的卷积模型。我们提出了两个扩展自我注意力,结合更有效的自我注意力的实现,提高这些模型的速度,内存使用和准确性。我们利用这些改进来开发一个新的自注意力模型家族HaloNets,它在ImageNet分类基准的参数限制设置上达到了最先进的精度。在初步的迁移学习实验中,我们发现HaloNet模型优于更大的模型,并且具有更好的推理性能。在诸如目标检测和实例分割等更难的任务上,我们简单的局部自注意和卷积混合算法在非常强的基线上显示出了改进。这些结果标志着在证明自我注意力模型在传统上由卷积模型主导的设置上的有效性方面又迈出了一步。

英文摘要:

Self-attention has the promise of improving computer vision systems due to parameter-independent scaling of receptive fields and content-dependent interactions, in contrast to parameter-dependent scaling and content-independent interactions of convolutions. Self-attention models have recently been shown to have encouraging improvements on accuracy-parameter trade-offs compared to baseline convolutional models such as ResNet-50. In this work, we aim to develop self-attention models that can outperform not just the canonical baseline models, but even the high-performing convolutional models. We propose two extensions to self-attention that, in conjunction with a more efficient implementation of self-attention, improve the speed, memory usage, and accuracy of these models. We leverage these improvements to develop a new self-attention model family, HaloNets, which reach state-of-the-art accuracies on the parameter-limited setting of the ImageNet classification benchmark. In preliminary transfer learning experiments, we find that HaloNet models outperform much larger models and have better inference performance. On harder tasks such as object detection and instance segmentation, our simple local self-attention and convolutional hybrids show improvements over very strong baselines. These results mark another step in demonstrating the efficacy of self-attention models on settings traditionally dominated by convolutional models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑