MambaDFuse：一种基于Mamba的多模态图像融合双阶段模型
MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion
AI总结：
针对现有多模态图像融合方法特征提取效率与效果受限的问题，提出基于Mamba的双阶段融合模型MambaDFuse，通过双层特征提取、双阶段融合及逆变换重建，在多类融合任务及下游检测任务中取得性能提升。
摘要：
多模态图像融合（MMIF）旨在将不同模态的互补信息整合到单张融合图像中，以全面表征成像场景并支撑下游视觉任务。近年来，随着深度神经网络的发展，多模态图像融合任务取得了显著进展，但现有方法受限于CNN固有的局部归纳偏置，或Transformer的二次计算复杂度，无法高效且有效地提取模态特有特征与模态融合特征。为解决这一问题，本文提出一种基于Mamba的双阶段融合（MambaDFuse）模型。首先，设计双层特征提取器，通过CNN块和Mamba块分别提取低层与高层特征，捕获单模态图像的长程依赖特征。随后，提出双阶段特征融合模块以获取融合不同模态互补信息的融合特征：该模块采用通道交换方法实现浅层融合，利用增强型多模态Mamba（M3）块实现深层融合。最后，融合图像重建模块通过特征提取的逆变换生成融合结果。大量实验表明，本文方法在红外-可见光图像融合和医学图像融合任务中均取得了优异的融合效果；此外，在统一基准测试中，MambaDFuse在目标检测等下游任务上也展现出更优的性能。代码与模型权重将在同行评审后公开。
英文摘要：
Multi-modality image fusion (MMIF) aims to integrate complementary information from different modalities into a single fused image to represent the imaging scene and facilitate downstream visual tasks comprehensively. In recent years, significant progress has been made in MMIF tasks due to advances in deep neural networks. However, existing methods cannot effectively and efficiently extract modality-specific and modality-fused features constrained by the inherent local reductive bias (CNN) or quadratic computational complexity (Transformers). To overcome this issue, we propose a Mamba-based Dual-phase Fusion (MambaDFuse) model. Firstly, a dual-level feature extractor is designed to capture long-range features from single-modality images by extracting low and high-level features from CNN and Mamba blocks. Then, a dual-phase feature fusion module is proposed to obtain fusion features that combine complementary information from different modalities. It uses the channel exchange method for shallow fusion and the enhanced Multi-modal Mamba (M3) blocks for deep fusion. Finally, the fused image reconstruction module utilizes the inverse transformation of the feature extraction to generate the fused result. Through extensive experiments, our approach achieves promising fusion results in infrared-visible image fusion and medical image fusion. Additionally, in a unified benchmark, MambaDFuse has also demonstrated improved performance in downstream tasks such as object detection. Code with checkpoints will be available after the peer-review process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
