arXiv 2409.126122024-09-19cs.CV

增强遥感图像变化描述中关键变化的感知

Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning

National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（武汉大学计算机学院国家多媒体软件工程技术研究中心）；Hubei Luojia Laboratory（湖北珞珈实验室）；School of Urban Design, Wuhan University（武汉大学城市设计学院）；School of Remote Sensing and Information Engineering, Wuhan University（武汉大学遥感信息工程学院）

Cong Yang, Zuchao Li, Hongzan Jiao, Zhi Gao, Lefei Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对遥感图像变化描述中无关特征干扰问题，提出KCFI多模态框架，通过关键变化特征引导和指令微调大语言模型，并联合像素级检测任务，在LEVIR-CC数据集上取得最优性能。

摘要：

近年来，尽管遥感图像变化描述已取得显著进展，但现有方法无法过滤掉与实际变化无关的区域，使模型易受无关特征影响。本文提出了一种新颖的多模态框架，用于遥感图像变化描述，由关键变化特征和指令微调（KCFI）引导。该框架旨在通过视觉指令充分利用大语言模型的内在知识，并利用像素级变化检测任务增强变化特征的有效性和准确性。具体而言，KCFI 包括一个用于提取双时相遥感图像特征的 ViTs 编码器、一个用于识别关键变化区域的关键特征感知器、一个用于约束关键变化特征的像素级变化检测解码器，以及一个基于大语言模型的指令微调解码器。此外，为确保变化描述和变化检测任务联合优化，我们采用动态权重平均策略来平衡两个任务之间的损失。我们还探索了用于视觉微调指令的各种特征组合，并证明仅使用关键变化特征来引导大语言模型是最优选择。为验证我们方法的有效性，我们在 LEVIR-CC 数据集上与几种最先进的变化描述方法进行了比较，取得了最佳性能。我们的代码将在 https://github.com/yangcong356/KCFI.git 上提供。

英文摘要：

Recently, while significant progress has been made in remote sensing image change captioning, existing methods fail to filter out areas unrelated to actual changes, making models susceptible to irrelevant features. In this article, we propose a novel multimodal framework for remote sensing image change captioning, guided by Key Change Features and Instruction-tuned (KCFI). This framework aims to fully leverage the intrinsic knowledge of large language models through visual instructions and enhance the effectiveness and accuracy of change features using pixel-level change detection tasks. Specifically, KCFI includes a ViTs encoder for extracting bi-temporal remote sensing image features, a key feature perceiver for identifying critical change areas, a pixel-level change detection decoder to constrain key change features, and an instruction-tuned decoder based on a large language model. Moreover, to ensure that change description and change detection tasks are jointly optimized, we employ a dynamic weight-averaging strategy to balance the losses between the two tasks. We also explore various feature combinations for visual fine-tuning instructions and demonstrate that using only key change features to guide the large language model is the optimal choice. To validate the effectiveness of our approach, we compare it against several state-of-the-art change captioning methods on the LEVIR-CC dataset, achieving the best performance. Our code will be available at https://github.com/yangcong356/KCFI.git.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。