arXiv 2410.008712025-05-23cs.CVcs.AI

MAP：释放混合Mamba-Transformer视觉骨干网络的潜力，通过掩码自回归预训练

MAP: Unleashing Hybrid Mamba-Transformer Vision Backbone's Potential with Masked Autoregressive Pretraining

IIIS, Tsinghua University（清华大学人工智能研究院）；Shanghai Qi Zhi Institute（上海启智研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

Yunze Liu, Li Yi

发表信息：: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对混合Mamba-Transformer视觉骨干网络，提出掩码自回归预训练（MAP），结合MAE与自回归优势，统一提升Mamba和Transformer模块性能，在2D和3D数据集上达到最先进水平。

摘要：

混合Mamba-Transformer网络近来引起了广泛关注。这些网络能够利用Transformer的可扩展性，同时发挥Mamba在长上下文建模和计算效率方面的优势。然而，如何有效预训练此类混合网络仍是一个悬而未决的问题。现有方法，如掩码自编码器（MAE）或自回归（AR）预训练，主要聚焦于单一类型的网络架构。相比之下，针对混合架构的预训练策略必须同时对Mamba和Transformer组件有效。基于此，我们提出掩码自回归预训练（MAP）来预训练混合Mamba-Transformer视觉骨干网络。该策略结合了MAE和自回归预训练的优势，在统一范式内提升了Mamba和Transformer模块的性能。实验结果表明，使用MAP预训练的混合Mamba-Transformer视觉骨干网络显著优于其他预训练策略，达到了最先进的性能。我们在2D和3D数据集上验证了该方法的有效性，并提供了详细的消融研究以支持每个组件的设计选择。代码和检查点可在https://github.com/yunzeliu/MAP获取。

英文摘要：

Hybrid Mamba-Transformer networks have recently garnered broad attention. These networks can leverage the scalability of Transformers while capitalizing on Mamba's strengths in long-context modeling and computational efficiency. However, the challenge of effectively pretraining such hybrid networks remains an open question. Existing methods, such as Masked Autoencoders (MAE) or autoregressive (AR) pretraining, primarily focus on single-type network architectures. In contrast, pretraining strategies for hybrid architectures must be effective for both Mamba and Transformer components. Based on this, we propose Masked Autoregressive Pretraining (MAP) to pretrain a hybrid Mamba-Transformer vision backbone network. This strategy combines the strengths of both MAE and Autoregressive pretraining, improving the performance of Mamba and Transformer modules within a unified paradigm. Experimental results show that the hybrid Mamba-Transformer vision backbone network pretrained with MAP significantly outperforms other pretraining strategies, achieving state-of-the-art performance. We validate the method's effectiveness on both 2D and 3D datasets and provide detailed ablation studies to support the design choices for each component. The code and checkpoints are available at https://github.com/yunzeliu/MAP

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。