面向遥感图像变化检索与字幕生成的多模态框架研究
Towards a multimodal framework for remote sensing image change retrieval and captioning
AI总结：
针对现有遥感数据多模态研究忽略其时序变化特性的不足，提出基于对比学习和LEVIR-CC数据集的双时相遥感图像基础模型，兼具图文检索与媲美SOTA的字幕生成能力并开源相关资源。
摘要：
近年来，人们对将文本与图像、音频、视频等其他模态融合的多模态应用兴趣日益浓厚，这类应用旨在推动与多模态AI系统的自然语言交互。虽然涉及标准模态的应用已被广泛探索，但针对遥感（RS）数据这类特定数据模态的研究仍较为匮乏。尽管遥感数据在环境保护、灾害监测、土地规划等领域有众多潜在应用，但现有解决方案大多聚焦于分类、字幕生成、检索等特定任务，往往忽略了遥感数据的独特特性，比如它能够随时间推移系统性地提供同一地理区域的信息，这一能力可实现对地表景观变化的持续监测。为填补这一空白，我们在变化检测分析的背景下，针对双时相遥感图像对提出了一种新型基础模型，该模型利用对比学习和LEVIR-CC数据集，同时实现字幕生成和图文检索功能。通过联合训练对比编码器和字幕生成解码器，我们的模型在双时相变化检测场景下新增了图文检索能力，同时保持了与当前最优水平相当的字幕生成性能。我们已在以下地址开源代码与预训练权重：https://github.com/rogerferrod/RSICRC。
英文摘要：
Recently, there has been increasing interest in multimodal applications that integrate text with other modalities, such as images, audio and video, to facilitate natural language interactions with multimodal AI systems. While applications involving standard modalities have been extensively explored, there is still a lack of investigation into specific data modalities such as remote sensing (RS) data. Despite the numerous potential applications of RS data, including environmental protection, disaster monitoring and land planning, available solutions are predominantly focused on specific tasks like classification, captioning and retrieval. These solutions often overlook the unique characteristics of RS data, such as its capability to systematically provide information on the same geographical areas over time. This ability enables continuous monitoring of changes in the underlying landscape. To address this gap, we propose a novel foundation model for bi-temporal RS image pairs, in the context of change detection analysis, leveraging Contrastive Learning and the LEVIR-CC dataset for both captioning and text-image retrieval. By jointly training a contrastive encoder and captioning decoder, our model add text-image retrieval capabilities, in the context of bi-temporal change detection, while maintaining captioning performances that are comparable to the state of the art. We release the source code and pretrained weights at: https://github.com/rogerferrod/RSICRC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
