In recent years, the application of multimodal large language models (MLLM) in various fields has achieved remarkable success. However, as the foundation model for many downstream tasks, current MLLMs are composed of the well-known Transformer network, which has a less efficient quadratic computation complexity. To improve the efficiency of such basic models, we propose Cobra, a linear computational complexity MLLM. Specifically, Cobra integrates the efficient Mamba language model into the visual modality. Moreover, we explore and study various modal fusion schemes to create an effective multi-modal Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely competitive performance with current computationally efficient state-of-the-art methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster speed due to Cobra's linear sequential modeling. (2) Interestingly, the results of closed-set challenging prediction benchmarks show that Cobra performs well in overcoming visual illusions and spatial relationship judgments. (3) Notably, Cobra even achieves comparable performance to LLaVA with about 43% of the number of parameters. We will make all codes of Cobra open-source and hope that the proposed method can facilitate future research on complexity problems in MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Sun Yat-sen University（中山大学）
KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；Tianjin Central Hospital of Gynecology Obstetrics（天津市中心妇产科医院）；Department of Computer Science and Information Technology, La Trobe University（拉筹伯大学计算机科学与信息技术系）；CSIRO Data61（澳大利亚联邦科学与工业研究组织Data61）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Computer Software, Tianjin University（天津大学计算机软件学院）；Centre for Artificial Intelligence driven Drug Discovery, Faculty of Applied Science, Macao Polytechnic University（澳门理工大学应用科学学院人工智能驱动药物发现中心）；Department of Cardiology, Tianjin Chest Hospital（天津市胸科医院心内科）
CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering
School of Software, Northwestern Polytechnical University, Shaanxi, China（西北工业大学软件学院）；Yangtze River Delta Research Institute of Northwestern Polytechnical University, Taicang, China（西北工业大学长江三角研究 institute）；Department of Computer Science and Information Technology, La Trobe University, Melbourne, Australia（拉筹伯大学计算机科学与信息技术系）；CSIRO Data61, Sydney, Australia（CSIRO Data61）；School of Intelligence Science and Technology, Nanjing University, Suzhou, China（南京大学智能科学与技术学院）；Australian Institute of Health Innovation (AIHI), Macquarie University, Australia（麦考瑞大学健康创新研究所）；School of Computer Software, College of Intelligence and Computing, Tianjin University, Tianjin, China（天津大学计算机软件学院）；Centre for Artificial Intelligence driven Drug Discovery, Faculty of Applied Science, Macao Polytechnic University, Macao Special Administrative Region of China（澳门理工学院人工智能驱动药物发现中心）；Department of Computer Science, School of Engineering, Shantou University, Guangdong, China（汕头大学计算机科学系）
RemoteDet-Mamba: A Hybrid Mamba-CNN Network for Multi-modal Object Detection in Remote Sensing Images
School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, China（计算机科学学院，北京邮电大学）；School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing, China（电子工程学院，北京邮电大学）
Global and Local Mamba Network for Multi-Modality Medical Image Super-Resolution
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University（中南大学机器视觉与智能医学湖南省工程研究中心）；University of Rouen-Normandy（鲁昂-诺曼底大学）；Henri Becquerel Cancer Center（亨利·贝克勒尔癌症中心）
AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation
School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）；School of Innovation and Entrepreneurship, Dalian University of Technology（大连理工大学创新创业学院）；School of Artificial Intelligence, Dalian University of Technology（大连理工大学人工智能学院）
Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion
University of Science and Technology of China（中国科学技术大学）；Institute of Intelligent Machines（智能机器研究所）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；Intelligent Agriculture Engineering Laboratory of Anhui Province, Institute of Intelligent Machines（安徽省智能农业工程实验室，智能机器研究所）
MambaMorph: a Mamba-based Framework for Medical MR-CT Deformable Registration
Image Processing Center, Beihang University, Beijing, China（北京航空航天大学图像处理中心）；Robotics Institute, Beihang University, Beijing, China（北京航空航天大学机器人研究所）；Tongji Hospital, Huazhong University of Science and Technology, Wuhan, China（华中科技大学同济医院）
Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image
School of Computer Science and Engineering, Central South University, Changsha, 410083, China（计算机科学与工程学院，中南大学，长沙，410083，中国）；Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Central South University, Changsha, 410083, China（机器视觉与智能医学工程研究中心，中南大学，长沙，410083，中国）；Department of Radiation Oncology, Washington University in St. Louis, USA（放射肿瘤科，华盛顿大学圣路易斯分校，美国）；Department of Nuclear Medicine, Henri Becquerel Cancer Center, Rouen, France（核医学科，亨利·贝克勒尔癌症中心，鲁昂，法国）；University of Rouen-Normandy, AMIS - QuantIF UR 4108, F-76000, Rouen, France（鲁昂-诺曼底大学，AMIS - QuantIF UR 4108，法国，F-76000，鲁昂，法国）
DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing
University of Electronic Science and Technology of China（电子科技大学）；Beijing Institute of Space Mechanics and Electricity（北京空间机电研究所）；Donghai Laboratory（东海实验室）
Mamba-Based Modality Disentanglement Network for Multi-Contrast MRI Reconstruction
School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）
Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusion
School of Future Technology, School of Artificial Intelligence, Dalian University of Technology（大连理工大学未来技术学院、人工智能学院）；Affiliated Zhongshan Hospital of Dalian University（大连大学附属中山医院）；Central Hospital of Dalian University of Technology（大连理工大学中心医院）；School of Information and Communication Engineering, Dalian University of Technology（大连理工大学信息与通信工程学院）
MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusion
The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Stanford University（斯坦福大学）
TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Hong Kong Polytechnic University（香港理工大学）；Zhuhai Campus of Sun Yat-sen University（中山大学珠海校区）；University of Adelaide（阿德莱德大学）；Peking University（北京大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)（布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系）；CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)（布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所）
MambaFusion: Height-Fidelity Dense Global Fusion for Multi-modal 3D Object Detection
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（多模态人工智能系统国家重点实验室（MAIS），中国科学院自动化所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Anyverse Intelligence；Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information（北京多模态信息超级智能安全重点实验室）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
GFE-Mamba: Mamba-based AD Multi-modal Progression Assessment via Generative Feature Extraction from MCI
Hangzhou Dianzi University（杭州电子科技大学）；Beihang University（北京航空航天大学）；Xuanwu Hospital, Capital Medical University（首都医科大学宣武医院）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai 200032, China（复旦大学基础医学学院数字医学研究中心）；Shanghai Key Lab of Medical Image Computing（上海医学图像计算重点实验室）
H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Artificial Intelligence, University of Chinese Academy of Science（中国科学院大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong Polytechnic University（香港理工大学）
I2I-Mamba: Multi-modal medical image synthesis via selective state space modeling
Department of Electrical and Electronics Engineering, and National Magnetic Resonance Research Center, Bilkent University（电子工程系和国家磁共振研究中心，比尔肯特大学）；Department of Electrical Engineering, Stanford University（电气工程系，斯坦福大学）
Backbone-Conditional Behavior of Modality Gating in Multi-Modal Prostate MRI Segmentation: A 5-Fold Cross-Validation and Gate Mechanism Analysis
The Second Xiangya Hospital of Central South University（中南大学湘雅医学院第二医院）；The Third Xiangya Hospital of Central South University（中南大学湘雅医学院第三医院）；School of Life Sciences, Central South University（中南大学生命科学学院）；Hunan Provincial Key Laboratory of Medical Information Research (Central South University)（湖南省医学信息研究重点实验室（中南大学））；Hunan Provincial Clinical Medical Research Center for Cardiovascular Intelligent Medicine（湖南省心血管智能医学临床医学研究中心）
AMD-Mamba: A Phenotype-Aware Multi-Modal Framework for Robust AMD Prognosis
Department of Population Health Sciences, Weill Cornell Medicine（人口健康科学系，韦尔·柯林斯医学中心）；Department of Surgery, University of Minnesota（外科系，明尼苏达大学）；Department of Biomedical Informatics and Data Science, Yale School of Medicine（生物医学信息学与数据科学系，耶鲁医学院）；National Eye Institute, National Institutes of Health（国家眼科研究所，国立卫生研究院）；National Library of Medicine, National Institutes of Health（国家医学图书馆，国立卫生研究院）
Surgical-MambaLLM: Mamba2-enhanced Multimodal Large Language Model for VQLA in Robotic Surgery
Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Imperial College London（帝国理工学院伦敦分校）；The Hong Kong University of Science and Technology（香港科学与技术大学）
UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking
Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）；Guangxi Key Laboratory of Multi-Source Information Mining and Security, Guangxi Normal University（广西多源信息挖掘与安全重点实验室，广西师范大学）；University Engineering Research Center of Educational Intelligent Technology, Guangxi Normal University（教育智能技术大学工程研究中心，广西师范大学）；Guangxi Key Laboratory of Machine Vision and Intelligent Control, Wuzhou University（广西机器视觉与智能控制重点实验室，梧州大学）
Context-Gated Cross-Modal Perception with Visual Mamba for PET-CT Lung Tumor Segmentation
Unit of Artificial Intelligence and Computer Systems, Università Campus Bio-Medico di Roma, Italy（人工智能与计算机系统单位，罗马大学生物医学校园，意大利）；College of Computer Science and Software Engineering, Shenzhen University, China（计算机科学与软件工程学院，深圳大学，中国）；Dept. of Computer Engineering, Modeling, Electronic and System Engineering, University of Calabria, Italy（计算机工程、建模、电子与系统工程系，卡拉布里亚大学，意大利）；IRCCS San Raffaele Hospital, Italy（圣拉斐拉医院，意大利）；Faculty of Medicine, Vita-Salute San Raffaele University, Italy（医学学院，维塔-桑拉斐拉大学，意大利）；Dept. of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering, Umeå University, Sweden（诊断与干预系，辐射物理，生物医学工程，乌梅大学，瑞典）；Dept. of Naval, Electrical, Electronics and Telecommunications Engineering, University of Genoa, Italy（海军、电子、电子与电信工程系，热那亚大学，意大利）
Interactive Spatial-Frequency Fusion Mamba for Multi-Modal Image Fusion
School of Future Technology, Dalian University of Technology and the Key Laboratory of Data Science and Smart Education (Hainan Normal University), Ministry of Education（未来技术学院，大连理工大学和数据科学与智能教育关键实验室（海南师范大学），教育部）；Affiliated Zhongshan Hospital of Dalian University（大连大学附属中山医院）；School of Computer Science and Artificial Intelligence, Wuhan University of Technology（计算机科学与人工智能学院，武汉理工大学）；Central Hospital of Dalian University of Technology（大连理工大学中心医院）；School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）
Hybrid Transformer-Mamba for Weakly Supervised Volumetric Medical Segmentation
University of Western Australia（西澳大学）；Harry Perkins Institute of Medical Research（哈利·佩金斯医学研究所）；National Imaging Facility（国家成像设施）；Fiona Stanley Hospital（菲奥娜·斯蒂尔医院）；Victor Chang Cardiac Research Institute（维多利亚·张心脏研究中心）
