视觉中基于 Mamba 的自回归预训练
Autoregressive Pretraining with Mamba in Vision
AI总结：
本文提出面向视觉 Mamba 的自回归预训练方法 ARM，提升训练效率与 ImageNet 性能，并释放其向超大模型扩展的能力。
摘要：
视觉领域已经开始使用近期开发的状态空间模型 Mamba 作为一系列任务的新骨干网络。本文表明，通过自回归预训练可以显著增强 Mamba 的视觉能力，而这一方向此前尚未得到探索。在效率方面，自回归特性能够充分利用 Mamba 的单向循环结构，因此与掩码建模等其他训练策略相比，可实现更快的整体训练速度。在性能方面，自回归预训练使 Mamba 架构相比其监督训练对应模型获得明显更高的准确率，更重要的是，它成功释放了该架构向大型甚至超大型模型规模扩展的潜力。例如，通过自回归预训练，一个 base 尺寸的 Mamba 在 ImageNet 上达到 83.2% 的准确率，比其监督训练对应模型高出 2.0%；我们的 huge 尺寸 Mamba 是迄今为止最大的 Vision Mamba，在 ImageNet 上达到 85.0% 的准确率（在使用 384×384 输入进行微调时达到 85.5%），显著超过视觉领域所有其他 Mamba 变体。代码可在 https://github.com/OliverRensu/ARM 获取。
英文摘要：
The vision community has started to build with the recently developed state space model, Mamba, as the new backbone for a range of tasks. This paper shows that Mamba's visual capability can be significantly enhanced through autoregressive pretraining, a direction not previously explored. Efficiency-wise, the autoregressive nature can well capitalize on the Mamba's unidirectional recurrent structure, enabling faster overall training speed compared to other training strategies like mask modeling. Performance-wise, autoregressive pretraining equips the Mamba architecture with markedly higher accuracy over its supervised-trained counterparts and, more importantly, successfully unlocks its scaling potential to large and even huge model sizes. For example, with autoregressive pretraining, a base-size Mamba attains 83.2\% ImageNet accuracy, outperforming its supervised counterpart by 2.0\%; our huge-size Mamba, the largest Vision Mamba to date, attains 85.0\% ImageNet accuracy (85.5\% when finetuned with $384\times384$ inputs), notably surpassing all other Mamba variants in vision. The code is available at \url{https://github.com/OliverRensu/ARM}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
