基于SAM的语义与运动变化区域挖掘用于遥感变化描述
SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning
AI总结：
本文提出基于SAM模型的遥感变化描述方法，通过融合语义和运动变化区域信息，提升变化描述的准确性和性能。
摘要：
遥感变化描述是一项新兴且热门的研究任务，旨在用自然语言描述两个不同时间拍摄的遥感图像之间感兴趣内容的变化。现有方法通常使用CNN/Transformer提取图像的视觉表示或结合辅助任务来提升最终结果，但具有较弱的区域意识和有限的时间对齐能力。为了解决这些问题，本文探索使用SAM（Segment Anything Model）基础模型来提取区域级表示，并将感兴趣区域的知识注入描述框架中。具体来说，我们采用CNN/Transformer模型提取全局视觉特征，利用SAM基础模型来界定语义和运动级的变化区域，并利用专门构建的知识图谱提供感兴趣对象的信息。这些异构信息源通过交叉注意力机制融合，再通过Transformer解码器生成最终的自然语言描述。广泛的实验结果表明，我们的方法在多个广泛使用的基准数据集上均取得了最先进的性能。本文的源代码将在https://github.com/Event-AHU/SAM_ChangeCaptioning上发布。
英文摘要：
Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
