arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.09498cs.CV

FusionMamba:基于Mamba的多模态图像融合动态特征增强

FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba

Great Bay University(大湾区大学);The Hong Kong Polytechnic University(香港理工大学);Macao Polytechnic University(澳门理工大学)

Xinyu Xie, Yawen Cui, Tao Tan, Xubin Zheng, Zitong Yu

AI总结:

提出FusionMamba框架,通过集成动态卷积和通道注意力改进Mamba,并设计动态特征融合模块(DFFM),有效解决了多模态图像融合中全局建模与计算成本的平衡问题,在多项任务中达到SOTA性能。

摘要:

多模态图像融合旨在整合来自不同成像技术的信息,为下游视觉任务生成一幅全面且细节丰富的单一图像。基于局部卷积神经网络(CNN)的现有方法难以高效捕获全局特征,而基于Transformer的模型尽管擅长全局建模,但计算成本高昂。Mamba通过利用选择性结构化状态空间模型(S4)来有效处理长距离依赖,同时保持线性复杂度,从而解决了这些局限性。在本文中,我们提出了FusionMamba,这是一个新颖的动态特征增强框架,旨在克服CNN和视觉Transformer(ViTs)在计算机视觉任务中面临的挑战。该框架通过集成动态卷积和通道注意力机制改进了视觉状态空间模型Mamba,这不仅保留了其强大的全局特征建模能力,还大大减少了冗余并增强了局部特征的表现力。此外,我们开发了一个名为动态特征融合模块(DFFM)的新模块。它结合了用于纹理增强和差异感知的动态特征增强模块(DFEM),以及专注于增强模态间相关性同时抑制冗余信息的跨模态融合Mamba模块(CMFM)。实验表明,FusionMamba在多种多模态图像融合任务及下游实验中均取得了最先进的性能,证明了其广泛的适用性和优越性。

英文摘要:

Multimodal image fusion aims to integrate information from different imaging techniques to produce a comprehensive, detail-rich single image for downstream vision tasks. Existing methods based on local convolutional neural networks (CNNs) struggle to capture global features efficiently, while Transformer-based models are computationally expensive, although they excel at global modeling. Mamba addresses these limitations by leveraging selective structured state space models (S4) to effectively handle long-range dependencies while maintaining linear complexity. In this paper, we propose FusionMamba, a novel dynamic feature enhancement framework that aims to overcome the challenges faced by CNNs and Vision Transformers (ViTs) in computer vision tasks. The framework improves the visual state-space model Mamba by integrating dynamic convolution and channel attention mechanisms, which not only retains its powerful global feature modeling capability, but also greatly reduces redundancy and enhances the expressiveness of local features. In addition, we have developed a new module called the dynamic feature fusion module (DFFM). It combines the dynamic feature enhancement module (DFEM) for texture enhancement and disparity perception with the cross-modal fusion Mamba module (CMFM), which focuses on enhancing the inter-modal correlation while suppressing redundant information. Experiments show that FusionMamba achieves state-of-the-art performance in a variety of multimodal image fusion tasks as well as downstream experiments, demonstrating its broad applicability and superiority.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑