arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.14899cs.CV
收藏

CrossViT:用于图像分类的交叉注意多尺度视觉Transformer

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室)

Chun-Fu Chen, Quanfu Fan, Rameswar Panda

摘要:

与卷积神经网络相比,最近开发的视觉Transformer(ViT)在图像分类方面取得了令人满意的结果。受此启发,本文研究了如何在Transformer模型中学习多尺度特征表示用于图像分类。为此,我们提出了一种双分支Transformer来组合图像块(即,Transformer中的令牌)以产生更强的图像特征。我们的方法用两个不同计算复杂性的独立分支来处理小补丁和大补丁令牌,然后这些令牌通过注意力多次融合以互补。此外,为了减少计算,我们开发了一个简单而有效的令牌融合模块的基础上交叉注意,它使用一个单一的令牌为每个分支作为一个查询,以交换信息与其他分支。我们提出的交叉注意只需要线性时间的计算和内存的复杂性,而不是二次的时间,否则。大量的实验表明,我们的方法表现得更好,或与几个并发的视觉Transformer的工作,除了有效的CNN模型。例如,在ImageNet1K数据集上,通过一些架构更改,我们的方法比最近的DeiT的性能高出2%,FLOP和模型参数略有增加。我们的源代码和模型可在\url{https://github.com/IBM/CrossViT}上获得。

英文摘要:

The recently developed vision transformer (ViT) has achieved promising results on image classification compared to convolutional neural networks. Inspired by this, in this paper, we study how to learn multi-scale feature representations in transformer models for image classification. To this end, we propose a dual-branch transformer to combine image patches (i.e., tokens in a transformer) of different sizes to produce stronger image features. Our approach processes small-patch and large-patch tokens with two separate branches of different computational complexity and these tokens are then fused purely by attention multiple times to complement each other. Furthermore, to reduce computation, we develop a simple yet effective token fusion module based on cross attention, which uses a single token for each branch as a query to exchange information with other branches. Our proposed cross-attention only requires linear time for both computational and memory complexity instead of quadratic time otherwise. Extensive experiments demonstrate that our approach performs better than or on par with several concurrent works on vision transformer, in addition to efficient CNN models. For example, on the ImageNet1K dataset, with some architectural changes, our approach outperforms the recent DeiT by a large margin of 2\% with a small to moderate increase in FLOPs and model parameters. Our source codes and models are available at \url{https://github.com/IBM/CrossViT}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑