arXiv 2108.016842021-08-03cs.CV

具有渐进采样功能的视觉Transformer

Vision Transformer with Progressive Sampling

SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；University of Oxford（牛津大学）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）；The Chinese University of Hong Kong（香港中文大学）

Xiaoyu Yue, Shuyang Sun, Zhanghui Kuang, Meng Wei, Philip Torr, Wayne Zhang, Dahua Lin

备注：: Accepted to ICCV 2021

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

摘要：

具有强大的全局关系建模能力的Transformers最近被引入到基本的计算机视觉任务中。作为一个典型的例子，视觉Transformer（ViT）直接将纯Transformer架构应用于图像分类，通过简单地将图像分割成具有固定长度的标记，并使用Transformers来学习这些标记之间的关系。然而，这种朴素的标记化可能会破坏对象结构，将网格分配给不感兴趣的区域，如背景，并引入干扰信号。为了缓解上述问题，在本文中，我们提出了一个迭代和渐进的采样策略来定位判别区域。在每次迭代时，当前采样步骤的嵌入被馈送到Transformer编码器层中，并且预测一组采样偏移以更新下一步骤的采样位置。渐进采样是可微的。当与视觉Transformer相结合时，所获得的PS-ViT网络可以自适应地学习看哪里。所提出的PS-ViT是有效的和高效的。当在ImageNet上从头开始训练时，PS-ViT在前1精度方面比vanilla ViT高3.8%，参数少了大约$4\times$，FLOP少了$10\times$。代码可在https://github.com/yuexy/PS-ViT上获得。

英文摘要：

Transformers with powerful global relation modeling abilities have been introduced to fundamental computer vision tasks recently. As a typical example, the Vision Transformer (ViT) directly applies a pure transformer architecture on image classification, by simply splitting images into tokens with a fixed length, and employing transformers to learn relations between these tokens. However, such naive tokenization could destruct object structures, assign grids to uninterested regions such as background, and introduce interference signals. To mitigate the above issues, in this paper, we propose an iterative and progressive sampling strategy to locate discriminative regions. At each iteration, embeddings of the current sampling step are fed into a transformer encoder layer, and a group of sampling offsets is predicted to update the sampling locations for the next step. The progressive sampling is differentiable. When combined with the Vision Transformer, the obtained PS-ViT network can adaptively learn where to look. The proposed PS-ViT is both effective and efficient. When trained from scratch on ImageNet, PS-ViT performs 3.8% higher than the vanilla ViT in terms of top-1 accuracy with about $4\times$ fewer parameters and $10\times$ fewer FLOPs. Code is available at https://github.com/yuexy/PS-ViT.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

摘要：

英文摘要：

相关论文