WeakTr: 探索纯视觉变换器用于弱监督语义分割
WeakTr: Exploring Plain Vision Transformer for Weakly-supervised Semantic Segmentation
AI总结：
本文提出WeakTr方法，利用纯视觉变换器的多层多头自注意力图进行弱监督语义分割和CAM生成，通过自适应融合提升CAM质量，并提出基于ViT的梯度裁剪解码器实现高效重训练。
摘要：
Transformer在各种计算机视觉任务中取得了巨大成功，理解其工作机制至关重要。作为测试案例，弱监督语义分割（WSSS）和类别激活图（CAM）是分析视觉变换器（ViT）的重要任务。基于使用ImageNet分类预训练的纯ViT，我们发现多层多头自注意力图能为弱监督语义分割和CAM生成提供丰富多样的信息，例如不同的ViT注意力头关注不同的图像区域和物体类别。因此，我们提出了一种新的方法，端到端估计注意力头的重要性，其中自注意力图被自适应融合以产生高质量的CAM结果，这些结果倾向于具有更完整的物体。此外，我们提出了一种基于ViT的梯度裁剪解码器，用于高效有效地在线重训练。此外，梯度裁剪解码器能充分利用大规模预训练ViT中的知识，并具有可扩展性。所提出的纯变换器基于弱监督学习方法（WeakTr）在标准基准上取得了优越的WSSS性能，即在PASCAL VOC 2012验证集上达到78.5%的mIoU，在COCO 2014验证集上达到51.1%的mIoU。源代码和检查点可在https://github.com/hustvl/WeakTr获取。
英文摘要：
Transformer has been very successful in various computer vision tasks and understanding the working mechanism of transformer is important. As touchstones, weakly-supervised semantic segmentation (WSSS) and class activation map (CAM) are useful tasks for analyzing vision transformers (ViT). Based on the plain ViT pre-trained with ImageNet classification, we find that multi-layer, multi-head self-attention maps can provide rich and diverse information for weakly-supervised semantic segmentation and CAM generation, e.g., different attention heads of ViT focus on different image areas and object categories. Thus we propose a novel method to end-to-end estimate the importance of attention heads, where the self-attention maps are adaptively fused for high-quality CAM results that tend to have more complete objects. Besides, we propose a ViT-based gradient clipping decoder for online retraining with the CAM results efficiently and effectively. Furthermore, the gradient clipping decoder can make good use of the knowledge in large-scale pre-trained ViT and has a scalable ability. The proposed plain Transformer-based Weakly-supervised learning method (WeakTr) obtains the superior WSSS performance on standard benchmarks, i.e., 78.5% mIoU on the val set of PASCAL VOC 2012 and 51.1% mIoU on the val set of COCO 2014. Source code and checkpoints are available at https://github.com/hustvl/WeakTr.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
