arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.09681cs.CVcs.LG
收藏

XCiT:交叉协方差图像Transformers

XCiT: Cross-Covariance Image Transformers

Facebook AI(脸书人工智能);Inria(法国国家信息与自动化研究所);Sorbonne University(索邦大学)

Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, Hervé Jegou

摘要:

继在自然语言处理方面取得成功之后,Transformers最近在计算机视觉方面表现出了很大的希望。Transformers底层的自注意操作产生了所有标记(即单词或图像块)之间的全局交互,并使图像数据的灵活建模超越了卷积的局部交互。然而,这种灵活性带来了时间和内存的二次复杂性,阻碍了长序列和高分辨率图像的应用。我们提出了一个“转置”版本的自我注意力,跨功能通道,而不是令牌,其中的相互作用是基于关键字和查询之间的互协方差矩阵。由此产生的互协方差注意力(XCA)在令牌的数量上具有线性复杂度,并且允许高效地处理高分辨率图像。我们的互协方差图像Transformer(XCiT)是基于XCA构建的。它将传统Transformers的准确性与卷积架构的可扩展性相结合。我们通过报告多个视觉基准测试的出色结果来验证XCiT的有效性和通用性,包括ImageNet-1 k上的图像分类和自监督特征学习,COCO上的目标检测和实例分割,以及ADE 20 k上的语义分割。

英文摘要:

Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑