arXiv 2606.170202026-08-04cs.CVcs.AI

FusionRS: 用于双模态视觉-语言基础模型的大规模RGB-红外遥感数据集

FusionRS: A Large-Scale RGB-Infrared-Style Remote Sensing Dataset for Cross-Modal Vision-Language Learning

China University of Petroleum-Beijing at Karamay（中国石油大学（北京）克拉玛依校区）；University of Electronic Science and Technology of China（电子科技大学）；Tianjin University（天津大学）

Jiaju Han, Ben Zhang, Xuemeng Sun, Qike Zhang, Yuxian Dong, Dingyi Lu, Chengyin Hu, Luwei Yang, Fengyu Zhang, Yiwei Wei, Jiujiang Guo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对遥感视觉-语言模型缺乏红外数据的问题，提出首个大规模RGB-红外-文本数据集FusionRS，通过翻译RGB图像为红外风格并配以红外感知描述，训练双模态基础模型，提升RGB-红外对齐和双模态字幕生成性能。

摘要：

遥感视觉-语言模型推动了地球观测理解的发展，但现有工作大多集中于RGB图像，红外数据中的互补信息尚未得到充分探索。红外图像提供了独特的线索，包括热强度结构、物体边界和光照不变场景特征，这些可以丰富超越传统RGB观测的视觉-语言学习。然而，用于遥感视觉-语言建模的大规模RGB-红外-文本数据集仍然缺失。为填补这一空白，我们引入了FusionRS，这是首个专为遥感双模态视觉-语言学习设计的大规模RGB-红外-文本数据集。FusionRS通过将多样的公开RGB遥感图像翻译为红外风格对应物，形成对齐的RGB-IR图像对。每对图像都配有常规场景描述和红外感知描述，后者在保留语义内容的同时明确描述红外特有的视觉属性。基于FusionRS，我们训练了用于RGB-IR联合理解的双模态视觉-语言基础模型。我们首先训练CLIP风格的模型进行RGB-IR-文本对齐，然后微调生成式VLM用于双模态RGB-IR字幕生成。实验表明，与仅RGB和非红外感知训练设置相比，FusionRS改进了RGB-IR对齐、红外到文本检索和双模态字幕生成。消融研究进一步验证了红外感知描述对于加强红外-语言对齐至关重要，突显了模态特定文本监督对于更可扩展的RGB-红外遥感视觉-语言表示学习的重要性。

英文摘要：

Remote sensing vision-language models have advanced Earth observation, but available large-scale vision-language resources remain RGB-centered, leaving complementary infrared information underexplored. Infrared observations provide distinctive intensity structures, object boundaries, and illumination-invariant cues that complement conventional RGB imagery, yet large-scale RGB-infrared-text resources remain scarce. We introduce FusionRS, the first large-scale RGB-infrared-style-text dataset for controlled dual-modal remote sensing vision-language learning. It contains 600,000 spatially aligned pairs created by translating diverse public RGB remote sensing images into infrared-style counterparts. Each pair retains a conventional scene caption, and a curated subset adds 45,913 IR-aware captions describing observable intensity, contrast, texture, and structure while preserving scene semantics. We train CLIP-style models for RGB-infrared-style-text alignment and adapt a generative vision-language model with mixed task-conditioned caption supervision. Evaluation covers cross-modal retrieval, scaling and supervision ablations, sensor-captured transfer, and strictly held-out captioning and VQA. FusionRS substantially improves RGB-infrared-style alignment and infrared-to-text retrieval over RGB-only and non-IR-aware settings. Ablations show that IR-aware captions improve task-conditioned infrared description, demonstrating the value of modality-specific supervision. FusionRS provides a scalable foundation for controlled RGB-infrared remote sensing vision-language learning.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。