Semantic-CC：通过基础知识与语义引导提升遥感图像变化描述
Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance
AI总结：
提出Semantic-CC方法，利用基础模型知识和像素级语义引导，通过多任务架构与三阶段训练，在LEVIR数据集上提升遥感图像变化描述的准确性与全面性。
摘要：
遥感图像变化描述（RSICC）旨在用自然语言阐述双时相遥感图像中感兴趣目标的变化。鉴于当前RSICC方法在多时相和多空间场景中表达通用特征的局限性，以及在提供细粒度、稳健且精确的变化描述方面的不足，我们提出了一种基于基础知识和语义引导的新型变化描述（CC）方法，称为Semantic-CC。Semantic-CC通过利用基础模型的潜在知识，减轻了高泛化算法对大量标注的依赖，并在变化检测（CD）提供的像素级语义引导下，生成更全面、更准确的变化描述。具体而言，我们提出了一种基于SAM的双时相编码器用于双图像特征提取；一个多任务语义聚合颈部，用于促进异构任务之间的信息交互；一个直接的多尺度变化检测解码器，以提供像素级语义引导；以及一个基于大语言模型（LLM）的变化描述解码器，用于生成变化描述句子。此外，为确保CD和CC联合训练的稳定性，我们提出了一种三阶段训练策略，在不同阶段监督不同任务。我们在LEVIR-CC和LEVIR-CD数据集上验证了所提出的方法。实验结果证实了CD和CC的互补性，表明Semantic-CC能够生成更准确的变化描述，并在两个任务上均达到最优性能。
英文摘要：
Remote sensing image change captioning (RSICC) aims to articulate the changes in objects of interest within bi-temporal remote sensing images using natural language. Given the limitations of current RSICC methods in expressing general features across multi-temporal and spatial scenarios, and their deficiency in providing granular, robust, and precise change descriptions, we introduce a novel change captioning (CC) method based on the foundational knowledge and semantic guidance, which we term Semantic-CC. Semantic-CC alleviates the dependency of high-generalization algorithms on extensive annotations by harnessing the latent knowledge of foundation models, and it generates more comprehensive and accurate change descriptions guided by pixel-level semantics from change detection (CD). Specifically, we propose a bi-temporal SAM-based encoder for dual-image feature extraction; a multi-task semantic aggregation neck for facilitating information interaction between heterogeneous tasks; a straightforward multi-scale change detection decoder to provide pixel-level semantic guidance; and a change caption decoder based on the large language model (LLM) to generate change description sentences. Moreover, to ensure the stability of the joint training of CD and CC, we propose a three-stage training strategy that supervises different tasks at various stages. We validate the proposed method on the LEVIR-CC and LEVIR-CD datasets. The experimental results corroborate the complementarity of CD and CC, demonstrating that Semantic-CC can generate more accurate change descriptions and achieve optimal performance across both tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
