MagicFuse: 单图像融合用于视觉与语义增强
MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement
AI总结：
本文提出MagicFuse单图像融合框架，通过扩散模型生成跨光谱场景表示，实现视觉与语义的双重约束，实验表明其性能优于多模态融合方法。
摘要：
本文聚焦于一个高度实用的场景：在仅使用可见成像传感器的情况下，如何继续利用多模态图像融合的优势。为此，我们提出了一种新的单图像融合概念，将其扩展到知识层面。具体而言，我们开发了MagicFuse，一种新的单图像融合框架，能够从单个低质量可见图像中推导出全面的跨光谱场景表示。MagicFuse首先引入了基于扩散模型的内在光谱知识增强分支和跨光谱知识生成分支。它们分别挖掘在可见光谱中被掩盖的场景信息，并学习转移到红外光谱的热辐射分布模式。在此基础上，我们设计了一个多领域知识融合分支，整合这两个分支的扩散流的概率噪声，从而通过连续采样获得跨光谱场景表示。然后，我们施加了视觉和语义约束，确保该场景表示能够满足人类观察同时支持下游语义决策。大量实验表明，尽管仅依赖单个退化的可见图像，我们的MagicFuse在视觉和语义表示性能上与或优于多模态输入的最先进融合方法。代码已公开在https://github.com/zhayanping/MagicFuse。
英文摘要：
This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single-image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image. The code is publicly available at https://github.com/zhayanping/MagicFuse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
