arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.01236cs.CVcs.AIcs.LG
收藏

用于语义分割的多尺度高分辨率视觉Transformer

Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation

University of Texas at Austin(德克萨斯大学奥斯汀分校);Meta Reality Labs(Meta现实实验室)

Jiaqi Gu, Hyoukjun Kwon, Dilin Wang, Wei Ye, Meng Li, Yu-Hsin Chen, Liangzhen Lai, Vikas Chandra, David Z. Pan

摘要:

与基于卷积神经网络(CNN)的模型相比,视觉Transformer(ViTs)在计算机视觉任务上具有更优越的性能。然而,ViT主要是为生成单尺度低分辨率表示的图像分类而设计的,这使得诸如语义分割之类的密集预测任务对ViT具有挑战性。因此,我们提出了HRViT,它增强了ViTs学习语义丰富和空间精确的多尺度表示,通过集成高分辨率的多分支架构与ViTs。我们平衡模型的性能和效率的HRViT的各种分支块协同优化技术。具体来说,我们探索异构分支设计,减少线性层中的冗余,并增加注意力块增强表现力。这些方法使HRViT能够将语义分割的性能和效率的帕累托边界推向一个新的水平,正如我们对ADE 20 K和Cityscapes的评估结果所显示的那样。HRViT在ADE 20 K上实现了50.20%的mIoU,在Cityscapes上实现了83.16%的mIoU,超过了最先进的MiT和CSWin主干,平均提高了+1.78 mIoU,节省了28%的参数,降低了21%的FLOP,证明了HRViT作为语义分割的强大视觉主干的潜力。

英文摘要:

Vision Transformers (ViTs) have emerged with superior performance on computer vision tasks compared to convolutional neural network (CNN)-based models. However, ViTs are mainly designed for image classification that generate single-scale low-resolution representations, which makes dense prediction tasks such as semantic segmentation challenging for ViTs. Therefore, we propose HRViT, which enhances ViTs to learn semantically-rich and spatially-precise multi-scale representations by integrating high-resolution multi-branch architectures with ViTs. We balance the model performance and efficiency of HRViT by various branch-block co-optimization techniques. Specifically, we explore heterogeneous branch designs, reduce the redundancy in linear layers, and augment the attention block with enhanced expressiveness. Those approaches enabled HRViT to push the Pareto frontier of performance and efficiency on semantic segmentation to a new level, as our evaluation results on ADE20K and Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction, demonstrating the potential of HRViT as a strong vision backbone for semantic segmentation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑