RNN作为线性Transformer：对视觉Mamba模型表示潜力的深入探讨
RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
AI总结：
本文深入研究了视觉Mamba模型的表示潜力，揭示其与Softmax和线性注意力的关系，并通过定量评估证明其在长距离依赖建模中的能力，同时展示了其在ImageNet上的高性能表现。
摘要：
Mamba最近因其在视觉任务中的有效性而受到关注。然而，其在视觉领域中的底层机制仍不为人所知。在本工作中，我们系统地研究了Mamba的表示特性，并做出了三个主要贡献。首先，我们从理论上分析了Mamba与Softmax和线性注意力之间的关系，证实了Mamba可以被视为Softmax注意力的低秩近似，从而弥合了Softmax与线性形式之间的表示差距。其次，我们引入了一个新的二进制分割度量标准用于激活图评估，将定性评估扩展为定量测量，展示了Mamba模型长距离依赖关系的能力。第三，通过利用DINO进行自监督预训练，我们获得了比标准监督方法产生的激活图更清晰的激活图，突显了Mamba在可解释性方面的潜力。值得注意的是，我们的模型在ImageNet上实现了78.5%的线性探测准确率，突显了其强大的性能。我们希望这项工作能为未来对基于Mamba的视觉架构的研究提供有价值的见解。
英文摘要：
Mamba has recently garnered attention as an effective backbone for vision tasks. However, its underlying mechanism in visual domains remains poorly understood. In this work, we systematically investigate Mamba's representational properties and make three primary contributions. First, we theoretically analyze Mamba's relationship to Softmax and Linear Attention, confirming that it can be viewed as a low-rank approximation of Softmax Attention and thereby bridging the representational gap between Softmax and Linear forms. Second, we introduce a novel binary segmentation metric for activation map evaluation, extending qualitative assessments to a quantitative measure that demonstrates Mamba's capacity to model long-range dependencies. Third, by leveraging DINO for self-supervised pretraining, we obtain clearer activation maps than those produced by standard supervised approaches, highlighting Mamba's potential for interpretability. Notably, our model also achieves a 78.5 percent linear probing accuracy on ImageNet, underscoring its strong performance. We hope this work can provide valuable insights for future investigations of Mamba-based vision architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
