CCExpert：通过差异感知集成与基础数据集提升MLLM在遥感变化描述中的能力
CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset
AI总结：
本文提出CCExpert模型，通过设计差异感知集成模块捕获双时相图像多尺度差异，并构建包含20万对图像和120万条描述的CC-Foundation数据集，结合三阶段渐进式训练，在LEVIR-CC基准上取得81.80分，显著提升了遥感变化描述性能。
摘要：
遥感图像变化描述（RSICC）旨在为多时相遥感图像之间的地表变化生成自然语言描述，详细说明变化对象的类别、位置和动态（例如增加或消失）。许多当前方法试图利用多模态大语言模型（MLLMs）的长序列理解和推理能力来完成此任务。然而，由于缺乏全面的数据支持，这些方法常常改变MLLMs的核心特征传输路径，破坏模型内在知识并限制其在RSICC中的潜力。本文提出一种基于新型先进多模态大模型框架的模型CCExpert。首先，我们设计了差异感知集成模块，捕获双时相图像间的多尺度差异并将其融入原始图像上下文，从而增强差异特征的信噪比。其次，我们构建了名为CC-Foundation的高质量多样化数据集，包含20万对图像和120万条描述，为该领域的持续预训练提供充足数据支持。最后，我们采用三阶段渐进式训练过程，确保差异感知集成模块与预训练MLLM的深度融合。CCExpert在LEVIR-CC基准上取得了$S^*_m=81.80$的显著性能，大幅超越先前最先进方法。代码及部分数据集即将在https://github.com/Meize0729/CCExpert开源。
英文摘要：
Remote Sensing Image Change Captioning (RSICC) aims to generate natural language descriptions of surface changes between multi-temporal remote sensing images, detailing the categories, locations, and dynamics of changed objects (e.g., additions or disappearances). Many current methods attempt to leverage the long-sequence understanding and reasoning capabilities of multimodal large language models (MLLMs) for this task. However, without comprehensive data support, these approaches often alter the essential feature transmission pathways of MLLMs, disrupting the intrinsic knowledge within the models and limiting their potential in RSICC. In this paper, we propose a novel model, CCExpert, based on a new, advanced multimodal large model framework. Firstly, we design a difference-aware integration module to capture multi-scale differences between bi-temporal images and incorporate them into the original image context, thereby enhancing the signal-to-noise ratio of differential features. Secondly, we constructed a high-quality, diversified dataset called CC-Foundation, containing 200,000 image pairs and 1.2 million captions, to provide substantial data support for continue pretraining in this domain. Lastly, we employed a three-stage progressive training process to ensure the deep integration of the difference-aware integration module with the pretrained MLLM. CCExpert achieved a notable performance of $S^*_m=81.80$ on the LEVIR-CC benchmark, significantly surpassing previous state-of-the-art methods. The code and part of the dataset will soon be open-sourced at https://github.com/Meize0729/CCExpert.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
