arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.01595cs.CVcs.AI

RS-MoE：一种面向遥感图像 captioning 和视觉问答的混合专家视觉语言模型

RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering

China Academy of Electronics and Information Technology（中国电子科技集团公司电子科学研究院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology（哈尔滨工业大学）；New York University Abu Dhabi（纽约大学阿布扎比分校）

Hui Lin, Danfeng Hong, Shuhang Ge, Chuyao Luo, Kai Jiang, Hao Jin, Congcong Wen

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对传统遥感图像 captioning 方法生成描述不足的问题，本文提出首个面向遥感领域的混合专家视觉语言模型 RS-MoE，通过指令路由器引导专家模型并行处理子任务，经两阶段训练在 RSICap 数据集等上取得 state-of-the-art 性能且泛化性良好。

摘要：

遥感图像 captioning（RSIC）面临独特挑战，在实际应用中发挥关键作用。传统 RSIC 方法往往难以生成丰富多样的描述。近年来，随着视觉语言模型（VLMs）的发展，出现了将这些模型整合到遥感领域，并引入专门设计的描述性数据集以增强 VLM 训练的尝试。本文提出 RS-MoE，这是首个针对遥感领域定制的基于混合专家（Mixture of Expert）的 VLM。与传统 MoE 模型不同，RS-MoE 的核心是 MoE 模块，该模块集成了新颖的指令路由器（Instruction Router）和多个轻量级大语言模型（LLMs）作为专家模型。指令路由器旨在为每个对应 LLM 生成特定提示，引导它们聚焦于 RSIC 任务的不同方面。这种设计不仅使每个专家 LLM 能专注于任务的特定子集，从而提升生成 caption 的特异性和准确性，还通过促进子任务的并行处理提高了模型的可扩展性。此外，我们提出了两阶段训练策略来微调 RS-MoE 模型，以防止稀疏性导致的性能下降。我们使用所提出的训练策略在 RSICap 数据集上对模型进行了微调。在 RSICap 数据集上的实验结果，以及在未进行额外微调的其他传统数据集上的评估表明，我们的模型在生成精确且上下文相关的 caption 方面达到了 state-of-the-art 性能。值得注意的是，我们的 RS-MoE-1B 变体性能可与 13B VLMs 媲美，证明了我们模型设计的高效性。此外，我们的模型在遥感视觉问答（RSVQA）任务中持续取得 state-of-the-art 性能，展现出良好的泛化能力。

英文摘要：

Remote Sensing Image Captioning (RSIC) presents unique challenges and plays a critical role in applications. Traditional RSIC methods often struggle to produce rich and diverse descriptions. Recently, with advancements in VLMs, efforts have emerged to integrate these models into the remote sensing domain and to introduce descriptive datasets specifically designed to enhance VLM training. This paper proposes RS-MoE, a first Mixture of Expert based VLM specifically customized for remote sensing domain. Unlike traditional MoE models, the core of RS-MoE is the MoE Block, which incorporates a novel Instruction Router and multiple lightweight Large Language Models (LLMs) as expert models. The Instruction Router is designed to generate specific prompts tailored for each corresponding LLM, guiding them to focus on distinct aspects of the RSIC task. This design not only allows each expert LLM to concentrate on a specific subset of the task, thereby enhancing the specificity and accuracy of the generated captions, but also improves the scalability of the model by facilitating parallel processing of sub-tasks. Additionally, we present a two-stage training strategy for tuning our RS-MoE model to prevent performance degradation due to sparsity. We fine-tuned our model on the RSICap dataset using our proposed training strategy. Experimental results on the RSICap dataset, along with evaluations on other traditional datasets where no additional fine-tuning was applied, demonstrate that our model achieves state-of-the-art performance in generating precise and contextually relevant captions. Notably, our RS-MoE-1B variant achieves performance comparable to 13B VLMs, demonstrating the efficiency of our model design. Moreover, our model demonstrates promising generalization capabilities by consistently achieving state-of-the-art performance on the Remote Sensing Visual Question Answering (RSVQA) task.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑