Mamba is an effective state space model with linear computation complexity. It has recently shown impressive efficiency in dealing with high-resolution inputs across various vision tasks. In this paper, we reveal that the powerful Mamba model shares surprising similarities with linear attention Transformer, which typically underperform conventional Transformer in practice. By exploring the similarities and disparities between the effective Mamba and subpar linear attention Transformer, we provide comprehensive analyses to demystify the key factors behind Mamba's success. Specifically, we reformulate the selective state space model and linear attention within a unified formulation, rephrasing Mamba as a variant of linear attention Transformer with six major distinctions: input gate, forget gate, shortcut, no attention normalization, single-head, and modified block design. For each design, we meticulously analyze its pros and cons, and empirically evaluate its impact on model performance in vision tasks. Interestingly, the results highlight the forget gate and block design as the core contributors to Mamba's success, while the other four designs are less crucial. Based on these findings, we propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention. The resulting model outperforms various vision Mamba models in both image classification and high-resolution dense prediction tasks, while enjoying parallelizable computation and fast inference speed. Code is available at https://github.com/LeapLabTHU/MLLA.
NeuroMamba: Multi-Perspective Feature Interaction with Visual Mamba for Neuron Segmentation
State Key Laboratory of Brain Cognition and Brain-Inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences（脑认知与脑启发智能技术国家重点实验室，自动化研究所）
Diversity-enhanced Collaborative Mamba for Semi-supervised Medical Image Segmentation
School of Intelligence Science and Technology, Nanjing University（智能科学与技术学院、南京大学）；School of Electrical and Information Engineering, The University of Sydney（电气与信息工程学院、悉尼大学）
Mamba Guided Boundary Prior Matters: A New Perspective for Generalized Polyp Segmentation
University of Surrey（萨里大学）；Côte d’Azur University（蔚蓝海岸大学）；Malaviya National Institute of Technology Jaipur（贾伊普尔马拉维亚国立理工学院）；University of South Dakota（南达科他大学）
The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong Baptist University（香港浸会大学）
Self-attention mechanisms, especially multi-head self-attention (MSA), have achieved great success in many fields such as computer vision and natural language processing. However, many existing vision transformer (ViT) works simply inherent transformer designs from NLP to adapt vision tasks, while ignoring the fundamental difference between ``how MSA works in image and language settings''. Language naturally contains highly semantic structures that are directly interpretable by humans. Its basic unit (word) is discrete without redundant information, which readily supports interpretable studies…
Vivim: a Video Vision Mamba for Medical Video Segmentation
The Hong Kong University of Science and Technology, Guangzhou（香港科技大学（广州））；The University of Hong Kong（香港大学）；Guangdong Provincial People’s Hospital（广东省人民医院）；Institute of High Performance Computing, A*STAR（新加坡高性能计算研究院（新加坡科研局下属机构））；Hong Kong University of Science and Technology（香港科技大学）
Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation
School of Digital Industry, Jiangxi Normal University（江西师范大学数字产业学院）；School of Computer and Information Engineering, Jiangxi Normal University（江西师范大学计算机与信息工程学院）
DH-Mamba: Exploring Dual-domain Hierarchical State Space Models for MRI Reconstruction
School of Basic Medical Science, Fudan University（复旦大学基础医学院）；Academy of Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Shandong Computer Science Center (National Supercomputer Center in Jinan)（山东计算机科学中心（国家济南超级计算中心））
SpectralMamba: Efficient Mamba for Hyperspectral Image Classification
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Mathematics and Statistics, Southeast University（东南大学数学学院）；Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家信息与自动化研究所、法国国家科学研究中心、格勒诺布尔理工学院、约瑟夫·傅里叶大学）
Cosine-Normalized Attention for Hyperspectral Image Classification
SDAIA-KFUPM, Joint Research Center for Artificial Intelligence (JRCAI), King Fahd University of Petroleum and Minerals（SDAIA-KFUPM联合人工智能研究中心（JRCAI），法赫德国王石油矿产大学）；Institute of Software Development and Engineering, Innopolis University（软件开发和工程学院，因诺波利斯大学）
