arXiv 2412.207422024-12-30cs.CV

UniRS：通过视觉语言模型统一多时相遥感任务

UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Models

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Geographic Sciences, Hunan Normal University（湖南师范大学）；City University of Hong Kong（香港城市大学）

Yujie Li, Wenjia Xu, Guangzuo Li, Zijian Yu, Zhiwei Wei, Jiuniu Wang, Mugen Peng

备注：: 12 pages, 5 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

UniRS是首个统一多时相遥感任务的视觉语言模型，支持单图、双时相图像对和视频输入，通过统一视觉表示、变化提取模块和提示增强机制，在视觉问答、变化描述和视频场景分类等任务上取得最优性能。

摘要：

遥感影像与自然图像之间的域差异近年来受到广泛关注，视觉语言模型（VLMs）已在遥感多模态任务中展现出优异的泛化性能。然而，当前研究在探索遥感视觉语言模型如何处理不同类型的视觉输入方面仍存在局限。为弥合这一差距，我们提出了UniRS，这是首个在各种视觉输入类型上统一多时相遥感任务的视觉语言模型。UniRS支持单张图像、双时相图像对和视频作为输入，能够在统一框架内实现全面的遥感时序分析。我们采用统一的视觉表示方法，使模型能够接受各种视觉输入。针对双时相图像对任务，我们定制了一个变化提取模块，以进一步增强时空特征的提取。此外，我们设计了一种针对模型推理过程的提示增强机制，利用通用视觉语言模型的先验知识为UniRS提供线索。为促进多任务知识共享，该模型在混合数据集上进行联合微调。实验结果表明，UniRS在视觉问答、变化描述和视频场景分类等多种任务上均取得了最先进的性能，凸显了其在统一这些多时相遥感任务方面的通用性和有效性。我们的代码和数据集将很快发布。

英文摘要：

The domain gap between remote sensing imagery and natural images has recently received widespread attention and Vision-Language Models (VLMs) have demonstrated excellent generalization performance in remote sensing multimodal tasks. However, current research is still limited in exploring how remote sensing VLMs handle different types of visual inputs. To bridge this gap, we introduce \textbf{UniRS}, the first vision-language model \textbf{uni}fying multi-temporal \textbf{r}emote \textbf{s}ensing tasks across various types of visual input. UniRS supports single images, dual-time image pairs, and videos as input, enabling comprehensive remote sensing temporal analysis within a unified framework. We adopt a unified visual representation approach, enabling the model to accept various visual inputs. For dual-time image pair tasks, we customize a change extraction module to further enhance the extraction of spatiotemporal features. Additionally, we design a prompt augmentation mechanism tailored to the model's reasoning process, utilizing the prior knowledge of the general-purpose VLM to provide clues for UniRS. To promote multi-task knowledge sharing, the model is jointly fine-tuned on a mixed dataset. Experimental results show that UniRS achieves state-of-the-art performance across diverse tasks, including visual question answering, change captioning, and video scene classification, highlighting its versatility and effectiveness in unifying these multi-temporal remote sensing tasks. Our code and dataset will be released soon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。