Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority.
FusionMamba: Efficient Remote Sensing Image Fusion with State Space Model
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);University of Chinese Academy of Sciences (UCAS)(中国科学院大学);University of Electronic Science and Technology of China (UESTC)(电子科技大学);Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新研究院)
UAVD-Mamba: Deformable Token Fusion Vision Mamba for Multimodal UAV Detection
College of Mechanical and Vehicle Engineering, Hunan University(湖南大学机械与车辆工程学院);School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院)
MambaX: Image Super-Resolution with State Predictive Control
Southeast University(东南大学);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院航空信息研究所);College of Resources and Environment, University of Chinese Academy of Sciences(中国科学院大学资源与环境学院);School of Mathematics, Southeast University(东南大学数学学院);Department of Complexity Science and Engineering, Graduate School of Frontier Sciences, the University of Tokyo(东京大学前沿科学研究院复杂科学与工程部门);Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK(格勒诺布尔阿尔卑斯大学、Inria、CNRS、Grenoble INP、LJK)
ClinicalFMamba: Advancing Clinical Assessment using Mamba-based Multimodal Neuroimaging Fusion
TD Bank Group, Toronto, Canada(TD银行集团,加拿大);Department of Computer Science, University of Toronto, Toronto, Canada(多伦多大学计算机科学系,加拿大);Department of Medical Imaging, University of Toronto, Toronto, Canada(多伦多大学医学影像系,加拿大)
Task-Generalized Adaptive Cross-Domain Learning for Multimodal Image Fusion
Key Laboratory of Opto-Electronic Information Science and Technology of Jiangxi Province, Nanchang Hangkong University(江西省光电信息科学与技术重点实验室,南昌航空大学);ReLER, CCAI, Zhejiang University(ReLER、CCAI、浙江大学);College of Engineering, Anhui Agricultural University(安徽农业大学工程学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机科学与信息工程学院)
CAWM-Mamba: A unified model for infrared-visible image fusion and compound adverse weather restoration
School of Physics(物理学院);Optoelectronic Engineering, Foshan University, Foshan 528225, China(光学电子工程学院,佛山大学,佛山528225,中国);Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology, Foshan 528225, China(粤港澳联合智能微纳光电子技术实验室,佛山528225,中国);School of Mechanical Electronic(机械电子学院);Information Engineering, China University of Mining and Technology, Beijing 100083, China(信息工程学院,中国矿业大学,北京100083,中国)
DACESR: Degradation-Aware Conditional Embedding for Real-World Image Super-Resolution
School of Electrical and Information Engineering, Zhengzhou University of Light Industry(郑州轻工业大学电气与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Automation, Central South University(中南大学自动化学院);Intelligent Manufacturing Engineering at Machinery Technology Development Co.,Ltd.(机械技术发展有限公司智能制造工程);China Academy of Machinery Science and Technology Group Co.,Ltd.(中国机械科学与技术集团有限公司)
FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);School of Physics and Optoelectronic Engineering, Foshan University(佛山大学物理与光电工程学院);School of Instrumentation Science and Optelectronics Engineering, Beihang University(北航仪器科学与光电工程学院);School of Information Engineering and Automation, Kunming University of Science and Technology(昆明理工大学信息工程与自动化学院)
VL-Mamba: Exploring State Space Models for Multimodal Learning
Australian Institute for Machine Learning(澳大利亚机器学习研究所);The University of Adelaide(阿德莱德大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks
School of Physics and Optoelectronic Engineering(物理与光电工程学院);Guangdong-HongKong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic Technology(粤港澳联合智能微纳光电技术实验室);School of Information Engineering and Automation(信息工程与自动化学院)
本文提出MambaTrans,一种多模态融合图像模态翻译器,利用多模态大语言模型描述和语义分割掩码作为输入,通过Multi-Model State Space Block捕获跨模态依赖,在不调整预训练模型参数的情况下提升下游视觉任务性能。
HGP-Mamba: Integrating Histology and Generated Protein Features for Mamba-based Multimodal Survival Risk Prediction
Cancer Hospital of Dalian University of Technology, Shenyang, China(大连理工大学沈阳医院);School of Biomedical Engineering, Faculty of Medicine, Dalian University of Technology, Dalian, China(大连理工大学生物医学工程学院);Key Laboratory of Integrated Circuit and Biomedical Electronic System, Dalian University of Technology, Dalian, China(大连理工大学集成电路与生物医学电子系统重点实验室)
MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters
School of Information and Electronics, Beijing Institute of Technology(信息与电子学院,北京理工大学);Beijing Institute of Technology Chongqing Innovation Center(北京理工大学重庆创新中心);National Key Laboratory for Space-Born Intelligent Information Processing(空间智能信息处理国家级重点实验室);School of Automation, Beijing Institute of Technology(自动化学院,北京理工大学)
Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations
College of Big Data and Internet(大数据与互联网学院);Shenzhen Technology University(深圳技术大学);Institute of Advanced Computing and Digital Engineering(先进计算与数字工程研究所);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
ME-Mamba: Multi-Expert Mamba with Efficient Knowledge Capture and Fusion for Multimodal Survival Analysis
Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai 200032, China(复旦大学基础医学学院数字医学研究中心);Shanghai Key Lab of Medical Image Computing(上海医学图像计算重点实验室)
VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network
Beihang University(北航);Peng Cheng Laboratory(鹏城实验室);Capital University of Physical Education and Sports(首都体育学院);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University(复旦大学脑智能科学与技术研究院);School of Computer Science, Fudan University(复旦大学计算机科学技术学院)
COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection
School of Resource and Environmental Sciences, Wuhan University(武汉大学资源与环境科学学院);State Key Laboratory of Information Engineering in Surveying, Mapping, and Remote Sensing, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室);School of Mathematics and Statistics, University of Glasgow(格拉斯哥大学数学与统计学院);School of Geophysics and Geomatics, China University of Geoscience(中国地质大学(武汉)地球物理与空间信息学院)
An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion
Labcin - Nature Inspired Computing Lab, Federal University of Esp\'irito Santo, Vit\'oria, Brazil PPGI - Graduate Program in Computer Science, Federal University of Esp\'irito Santo, Vit\'oria, Brazil
MambaTrack: Exploiting Dual-Enhancement for Night UAV Tracking
Cooperative Medianet Innovation Center(合作 medi 创新中心);Shanghai Jiao Tong University(上海交通大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));CloudWalk Technology(云从科技);Shanghai AI Laboratory(上海人工智能实验室)
GFE-Mamba: Mamba-based AD Multi-modal Progression Assessment via Generative Feature Extraction from MCI
Hangzhou Dianzi University(杭州电子科技大学);Beihang University(北京航空航天大学);Xuanwu Hospital, Capital Medical University(首都医科大学宣武医院);Shenzhen Research Institute of Big Data(深圳大数据研究院)
TK-Mamba: Marrying KAN With Mamba for Text-Driven 3D Medical Image Segmentation
Zhejiang University(浙江大学);Research Center, National Certification Technology (Hangzhou) Co., Ltd(国家认证技术(杭州)有限公司研究部);Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);East China Normal University(华东师范大学)
MamFusion: Multi-Mamba with Temporal Fusion for Partially Relevant Video Retrieval
Hangzhou City University(杭州城市大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Hangzhou Normal University(杭州师范大学);Zhejiang Provincial Engineering Research Center for Real-Time SmartTech in Urban Security Governance(浙江省实时智能安防技术工程研究中心)
CrossLLM-Mamba: Multimodal State Space Fusion of LLMs for RNA Interaction Prediction
Department of Computer Science, University of Kentucky, Lexington, KY, USA(计算机科学系,肯塔基大学,路易斯维尔,KY,美国);Department of Mathematics, University of Kentucky, Lexington, KY, USA(数学系,肯塔基大学,路易斯维尔,KY,美国);Institute for Biomedical Informatics, University of Kentucky, Lexington, KY, USA(生物医学信息学研究所,肯塔基大学,路易斯维尔,KY,美国)
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)(布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系);CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)(布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所)
TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving
Faculty of Marine Science and Technology, Beijing Institute of Technology(海洋科学与技术学院,北京理工大学);State Key Laboratory of Intelligent Technology and Systems and Department of Computer Science and Technology, Tsinghua University(智能技术与系统国家重点实验室和计算机科学与技术系,清华大学);School of Engineering and Applied Science, Yale University(工程与应用科学学院,耶鲁大学);University of Toronto(多伦多大学);Beijing University of Chemical Technology(北京化工大学)
GraphMMP: A Graph Neural Network Model with Mutual Information and Global Fusion for Multimodal Medical Prognosis
Hangzhou Dianzi University(杭州电子科技大学);Shenzhen Polytechnic University(深圳职业技术大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);Yunnan University(云南大学);Shenzhen University(深圳大学)
Flood-DamageSense: Multimodal Mamba with Multitask Learning for Building Flood Damage Assessment using SAR Remote Sensing Imagery
Urban Resilience.AI Lab(城市韧性人工智能实验室);Zachry Department of Civil and Environmental Engineering(扎克里土木与环境工程系);Texas A&M University(德克萨斯A&M大学);College Station, TX(学院站, 德克萨斯)
AMF-MedIT: An Efficient Align-Modulation-Fusion Framework for Medical Image-Tabular Data
School of Electronics and Communication Engineering(电子与通信工程学院);Guangdong Provincial Key Laboratory of Advanced IntelliSense Technology(广东省先进智能感知技术重点实验室);Department of Radiology(放射科)
GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning
Automation & Control Institute, TU Wien(维也纳技术大学自动化与控制研究所);Austrian Institute of Technology (AIT) GmbH(奥地利技术研究院);MBZUAI(穆罕默德·本·扎耶德人工智能大学);University of Liverpool(利物浦大学)
Selective Structured State Space for Multispectral-fused Small Target Detection
Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学);National Space Science Center, Chinese Academy of Sciences(中国科学院国家空间科学中心);School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);School of Astronomy and Space Science, University of Chinese Academy of Sciences(中国科学院大学天文与空间科学学院);School of Computing, National University of Singapore(新加坡国立大学计算机学院)
MMSF: Multitask and Multimodal Supervised Framework for WSI Classification and Survival Analysis
University of Chinese Academy of Sciences(中国科学院大学);Shanghai Advanced Research Institute, Chinese Academy of Sciences(中国科学院上海先进研究院);Department of Radiology, Changhai Hospital(昌海医院放射科)
A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond
Indian Institute of Technology Indore, India;R.M.D. Engineering College, Kavaraipettai, India;Jio Platforms Limited, India;Birla Institute of Technology \& Science Pilani, India