为什么 Mamba 有效？利用线性 Transformer-Mamba 网络进行多模态图像融合
Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion
AI总结：
本文提出 Tmamba 双分支网络，结合线性 Transformer 与 Mamba 实现线性复杂度下的全局建模，并通过 T-M 交互和跨模态注意力机制提升红外-可见光及医学图像融合性能。
摘要：
多模态图像融合旨在整合来自不同来源图像的优点，并生成高质量的融合图像。然而，现有的特征提取和融合方法要么受到推理过程中固有的局部感受野偏置和静态参数的限制（CNN），要么受到二次计算复杂度的限制（Transformer），无法有效地提取和融合特征。为了解决这一问题，我们提出了一种名为 Tmamba 的双分支图像融合网络。它由线性 Transformer 和 Mamba 组成，在保持线性复杂度的同时具备全局建模能力。由于 Transformer 和 Mamba 结构之间的差异，两个分支提取的特征分别携带通道信息和位置信息。在两个分支之间设计了 T-M 交互结构，分别使用全局可学习参数和卷积层来传递位置信息和通道信息。我们进一步在注意力层面提出了跨模态交互，以获得跨模态注意力。实验表明，我们的 Tmamba 在多个融合任务中取得了有前景的结果，包括红外-可见光图像融合和医学图像融合。代码及检查点将在同行评审过程后提供。
英文摘要：
Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
