TeSG：红外与可见光图像融合的文本语义引导
TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusion
AI总结：
针对红外与可见光图像融合中文本语义利用不足的问题，提出TeSG方法，通过掩码与文本双层语义引导，结合三个模块优化融合，提升下游任务性能。
摘要：
红外与可见光图像融合（IVF）旨在结合两种图像模态的互补信息，生成信息更丰富、更全面的输出。近年来，文本引导的IVF因其灵活性和通用性展现出巨大潜力。然而，文本语义信息的有效整合与利用仍研究不足。为应对这些挑战，我们在两个层面引入文本语义：掩码语义层面和文本语义层面，两者均源自大型视觉语言模型（VLMs）提取的文本描述。在此基础上，我们提出用于红外与可见光图像融合的文本语义引导方法，称为TeSG，该方法以针对下游任务（如检测和分割）优化的方式引导图像合成过程。具体而言，TeSG由三个核心组件构成：语义信息生成器（SIG）、掩码引导的交叉注意力（MGCA）模块和文本驱动的注意力融合（TDAF）模块。SIG基于文本描述生成掩码语义和文本语义。MGCA模块在掩码语义引导下，对红外和可见光图像的视觉特征进行基于注意力的初步融合。最后，TDAF模块利用文本语义驱动的门控注意力细化融合过程。大量实验表明，与现有最先进方法相比，我们的方法具有竞争力，尤其是在下游任务性能方面。
英文摘要：
Infrared and visible image fusion (IVF) aims to combine complementary information from both image modalities, producing more informative and comprehensive outputs. Recently, text-guided IVF has shown great potential due to its flexibility and versatility. However, the effective integration and utilization of textual semantic information remains insufficiently studied. To tackle these challenges, we introduce textual semantics at two levels: the mask semantic level and the text semantic level, both derived from textual descriptions extracted by large Vision-Language Models (VLMs). Building on this, we propose Textual Semantic Guidance for infrared and visible image fusion, termed TeSG, which guides the image synthesis process in a way that is optimized for downstream tasks such as detection and segmentation. Specifically, TeSG consists of three core components: a Semantic Information Generator (SIG), a Mask-Guided Cross-Attention (MGCA) module, and a Text-Driven Attentional Fusion (TDAF) module. The SIG generates mask and text semantics based on textual descriptions. The MGCA module performs initial attention-based fusion of visual features from both infrared and visible images, guided by mask semantics. Finally, the TDAF module refines the fusion process with gated attention driven by text semantics. Extensive experiments demonstrate the competitiveness of our approach, particularly in terms of performance on downstream tasks, compared to existing state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
