FS-Diff：语义引导与清晰度感知的多模态图像融合与超分辨率同步方法
FS-Diff: Semantic guidance and clarity-aware simultaneous multimodal image fusion and super-resolution
AI总结：
FS-Diff提出一种语义引导与清晰度感知的联合图像融合与超分辨率方法，通过条件生成和双向Mamba提取全局特征，在多个数据集上优于现有方法，恢复更丰富细节与语义。
摘要：
作为一种有影响力的信息融合和低级视觉技术，图像融合整合来自源图像的互补信息，以生成信息丰富的融合图像。近年来，已有一些尝试联合实现图像融合和超分辨率。然而，在军事侦察和远距离探测任务等实际应用中，多模态图像中的目标和背景结构容易受到损坏，分辨率低且语义信息弱，这导致当前融合技术的结果欠佳。为此，我们提出FS-Diff，一种语义引导和清晰度感知的联合图像融合与超分辨率方法。FS-Diff将图像融合和超分辨率统一为一个条件生成问题。它利用所提出的清晰度感知机制提供的语义引导，进行自适应低分辨率感知和跨模态特征提取。具体而言，我们将期望的融合结果初始化为纯高斯噪声，并引入双向特征Mamba来提取多模态图像的全局特征。此外，利用源图像和语义作为条件，我们通过改进的U-Net网络实现随机迭代去噪过程。该网络经过训练，可在多个噪声水平下进行去噪，以生成具有跨模态特征和丰富语义信息的高分辨率融合结果。我们还构建了一个强大的航空视图多场景（AVMS）基准，涵盖600对图像。在六个公开数据集和我们的AVMS数据集上进行的广泛联合图像融合和超分辨率实验表明，FS-Diff在多种放大倍数下优于最先进的方法，并且可以在融合图像中恢复更丰富的细节和语义。代码可在https://github.com/XylonXu01/FS-Diff获取。
英文摘要：
As an influential information fusion and low-level vision technique, image fusion integrates complementary information from source images to yield an informative fused image. A few attempts have been made in recent years to jointly realize image fusion and super-resolution. However, in real-world applications such as military reconnaissance and long-range detection missions, the target and background structures in multimodal images are easily corrupted, with low resolution and weak semantic information, which leads to suboptimal results in current fusion techniques. In response, we propose FS-Diff, a semantic guidance and clarity-aware joint image fusion and super-resolution method. FS-Diff unifies image fusion and super-resolution as a conditional generation problem. It leverages semantic guidance from the proposed clarity sensing mechanism for adaptive low-resolution perception and cross-modal feature extraction. Specifically, we initialize the desired fused result as pure Gaussian noise and introduce the bidirectional feature Mamba to extract the global features of the multimodal images. Moreover, utilizing the source images and semantics as conditions, we implement a random iterative denoising process via a modified U-Net network. This network istrained for denoising at multiple noise levels to produce high-resolution fusion results with cross-modal features and abundant semantic information. We also construct a powerful aerial view multiscene (AVMS) benchmark covering 600 pairs of images. Extensive joint image fusion and super-resolution experiments on six public and our AVMS datasets demonstrated that FS-Diff outperforms the state-of-the-art methods at multiple magnifications and can recover richer details and semantics in the fused images. The code is available at https://github.com/XylonXu01/FS-Diff.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
