Beihang University(北京航空航天大学);Johns Hopkins University(约翰斯·霍普金斯大学);Hangzhou Innovation Institute of Beihang University(北京航空航天大学杭州创新研究院);Zhengzhou University(郑州大学);University of Science and Technology of China(中国科学技术大学);Xidian University(西安电子科技大学)
The past year has witnessed the rapid development of applying the Transformer module to vision problems. While some researchers have demonstrated that Transformer-based models enjoy a favorable ability of fitting data, there are still growing number of evidences showing that these models suffer over-fitting especially when the training data is limited. This paper offers an empirical study by performing step-by-step operations to gradually transit a Transformer-based model to a convolution-based model. The results we obtain during the transition process deliver useful messages for improving visual recognition. Based on these observations, we propose a new architecture named Visformer, which is abbreviated from the `Vision-friendly Transformer'. With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy, and the advantage becomes more significant when the model complexity is lower or the training set is smaller. The code is available at https://github.com/danczs/Visformer.
MicroViTv2: Beyond the FLOPS for Edge Energy-Friendly Vision Transformers
Department of Electro-Optics, National Formosa University(国立.formosa大学电光学系);Department of Electrical Engineering, National Taipei University(台北国立大学电气工程系);College of Artificial Intelligence and Green Energy, National Yang Ming Chiao Tung University(阳明交通大学人工智能与再生能源学院)
DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
Hardware-Friendly Static Quantization Method for Video Diffusion Transformers
Division of the Humanities and Social Sciences(人文与社会科学系);California Institute of Technology(加州理工学院);Device Solutions Research America(设备解决方案研究美国);Samsung Semiconductor Inc.(三星半导体公司)
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization
Academy for Engineering and Technology, Fudan University(复旦大学工程与应用技术研究院);NVIDIA Corporation(英伟达公司);School for Information Science and Technology, Fudan University(复旦大学信息科学与工程学院)
Rethinking Local Perception in Lightweight Vision Transformer
Tsinghua University(清华大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
University of Texas at Austin(德克萨斯大学奥斯汀分校);UC Santa Barbara(加州大学圣塔芭芭拉分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室)
最近对Transformers的爆炸性兴趣表明它们有潜力成为计算机视觉任务的强大“通用”模型,例如分类,检测和分割。虽然这些尝试主要研究判别模型,但我们在一些更困难的视觉任务上探索了Transformers,例如,生成对抗网络(GANs)。我们的目标是进行第一个试点研究,建立一个完全没有卷积的GAN,只使用纯粹的基于transformer的架构。我们的香草GAN架构,被称为transGAN,由一个内存友好的基于transformer的生成器,逐步增加特征分辨率,并相应地多尺度的搜索,以同时捕获语义上下文和低级别的纹理。在此基础上,我们引入了网格自关注的新模块,以进一步缓解内存瓶颈,从而将TransGAN扩展到高分辨率生成。我们还开发了一个独特的训练配方,包括一系列技术,可以减轻transGAN的训练不稳定性问题,如数据增强,修改归一化和相对位置编码。与当前使用卷积主干的最先进的GANs相比,我们最好的架构实现了极具竞争力的性能。具体来说,TransGAN在STL-10上设置了10.43的最新初始得分和18.28的FID,优于StyleGAN-V2。当涉及到更高分辨率(例如256 x 256)的生成任务时,例如在CelebA-HQ和LSUN-Church上,TransGAN继续生成具有高保真度和令人印象深刻的纹理细节的各种视觉示例。此外,我们深入研究基于transformer的生成模…
University of Macau(澳门大学);Nanjing University of Science and Technology(南京理工大学);Singapore Management University(新加坡管理大学);Shanghai Jiaotong University(上海交通大学)
MST: Masked Self-Supervised Transformer for Visual Representation
Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);SenseTime Research(商汤科技研究院);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
Transformer已被广泛应用于自然语言处理(NLP)中的自监督预训练,并取得了巨大的成功。然而,在视觉自监督学习中,它还没有得到充分的探索。同时,以往的方法只从全局的角度考虑高层特征和学习表示,可能无法将其转移到关注局部特征的下游密集预测任务中。在本文中,我们提出了一种新的Masked自监督Transformer方法命名为MST,它可以显式地捕获图像的局部上下文,同时保留全局语义信息。具体而言,受自然语言处理中的MLM(Masked Language Modeling)思想的启发,提出了一种基于多头自注意映射的Masked token策略,该策略在不破坏自监督学习关键结构的前提下,动态地屏蔽局部块的部分token。更重要的是,屏蔽令牌连同剩余令牌一起由全局图像解码器进一步恢复,该全局图像解码器保留了图像的空间信息,并且对下游密集预测任务更友好。在多个数据集上的实验证明了该方法的有效性和通用性。例如,MST使用DeiT-S仅使用300个epoch预训练通过线性评估实现了76.9%的Top-1准确率,这比具有相同epoch的监督方法高出0.4%,比其可比变体DINO高出1.0%。对于密集预测任务,MST还在MS COCO目标检测上实现了42.7%的mAP,在Cityscapes分割上实现了74.04%的mIoU,仅使用100个epoch的预训练。
AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学虚拟现实技术与系统国家重点实验室);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院)
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Nationa Key Laboratory for Novel Software Technology(新型软件技术国家实验室);School of Artificial Intelligence(人工智能学院);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学)
RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)