MMMamba: 一种多功能跨模态在上下文融合框架用于全色锐化和零样本图像增强
MMMamba: A Versatile Cross-Modal In Context Fusion Framework for Pan-Sharpening and Zero-Shot Image Enhancement
AI总结：
MMMamba通过跨模态在上下文融合框架实现全色锐化和零样本图像增强，结合Mamba架构和多模态交错扫描机制，提升跨模态交互能力与计算效率。
摘要：
全色锐化旨在通过将高分辨率全色（PAN）图像与其对应的低分辨率多谱段（MS）图像融合，生成高分辨率多谱段（HRMS）图像。为了实现有效的融合，必须充分利用两种模态之间的互补信息。传统的基于CNN的方法通常依赖于通道级拼接和固定卷积运算符，这限制了它们适应不同空间和光谱变化的能力。虽然跨注意力机制能够实现全局交互，但它们计算效率低下，可能会稀释细粒度对应关系，使得难以捕捉复杂的语义关系。最近在多模态扩散变压器（MMDiT）架构方面的进展在图像生成和编辑任务中表现出色。与跨注意力机制不同，MMDiT采用在上下文中的条件化来促进更直接和高效的跨模态信息交换。在本文中，我们提出了MMMamba，一种用于全色锐化的跨模态在上下文融合框架，具有支持以零样本方式进行图像超分辨率的灵活性。基于Mamba架构，我们的设计确保了线性计算复杂度，同时保持了强大的跨模态交互能力。此外，我们引入了一种新的多模态交错（MI）扫描机制，以促进PAN和MS模态之间的有效信息交换。广泛的实验表明，与现有最先进（SOTA）技术相比，我们的方法在多个任务和基准上都表现出优越的性能。
英文摘要：
Pan-sharpening aims to generate high-resolution multispectral (HRMS) images by integrating a high-resolution panchromatic (PAN) image with its corresponding low-resolution multispectral (MS) image. To achieve effective fusion, it is crucial to fully exploit the complementary information between the two modalities. Traditional CNN-based methods typically rely on channel-wise concatenation with fixed convolutional operators, which limits their adaptability to diverse spatial and spectral variations. While cross-attention mechanisms enable global interactions, they are computationally inefficient and may dilute fine-grained correspondences, making it difficult to capture complex semantic relationships. Recent advances in the Multimodal Diffusion Transformer (MMDiT) architecture have demonstrated impressive success in image generation and editing tasks. Unlike cross-attention, MMDiT employs in-context conditioning to facilitate more direct and efficient cross-modal information exchange. In this paper, we propose MMMamba, a cross-modal in-context fusion framework for pan-sharpening, with the flexibility to support image super-resolution in a zero-shot manner. Built upon the Mamba architecture, our design ensures linear computational complexity while maintaining strong cross-modal interaction capacity. Furthermore, we introduce a novel multimodal interleaved (MI) scanning mechanism that facilitates effective information exchange between the PAN and MS modalities. Extensive experiments demonstrate the superior performance of our method compared to existing state-of-the-art (SOTA) techniques across multiple tasks and benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
