通过MLLM与LLM生成高质量图像-文本数据集增强遥感视觉-语言模型
Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation
AI总结：
本文提出MpGI两阶段方法，利用MLLM和LLM从多视角生成并整合高质量遥感图像描述，构建HQRS-IT-210K数据集，微调得到HQRS-CLIP和RS-CoCa模型，在极少训练数据下超越现有最先进方法。
摘要：
视觉-语言基础模型（VLFMs）在遥感（RS）影像中的应用因其在各种下游任务中的卓越能力而受到广泛关注。一个关键挑战在于缺乏高质量、大规模、图像-文本配对的训练数据。最近，一些工作引入了大规模的遥感图像-文本数据集并训练了他们的VLFMs。然而，由于用于生成描述文本的方法较为初级，数据集的质量不够理想，需要更大规模的训练数据，却只能获得有限的性能提升。在本文中，我们提出了一种名为MpGI（多视角生成与集成）的两阶段方法，用于为遥感图像生成高质量的文本描述。首先，我们利用Rule-MLLM（多模态大语言模型）中继生成和MLLMs生成方法，从不同视角生成独特且详细的描述。接下来，我们利用大语言模型（LLMs）将这些多样化的描述整合成全面的文本描述，捕捉来自多个视角的细节。最后，我们构建了HQRS-IT-210K数据集，包含约21万张遥感图像和130万条文本描述。我们使用该数据集对两个VLFMs进行了微调：判别式模型CLIP和图像到文本生成式模型CoCa。这一过程得到了我们提出的HQRS-CLIP和RS-CoCa模型。实验结果表明，HQRS-CLIP在仅使用4.2%训练数据的情况下，在多个下游任务中超越了此前最先进的遥感CLIP模型。RS-CoCa在基准数据集上优于其他先进方法，并且能够为遥感图像生成与人工标注相当甚至更优的文本描述。数据集、预训练模型和代码将在https://github.com/YiguoHe/HQRS-210K-and-HQRS-CLIP发布。
英文摘要：
The application of Vision-language foundation models (VLFMs) to remote sensing (RS) imagery has garnered significant attention due to their superior capability in various downstream tasks. A key challenge lies in the scarcity of high-quality, large-scale, image-text paired training data. Recently, several works introduced extensive image-text datasets for RS and trained their VLFMs. However, due to the rudimentary methods used for generating captions, the quality of datasets is suboptimal, requiring larger volumes of training data, while only yielding modest performance improvements. In this paper, we propose a two-stage method named MpGI(Multi-Perspective Generation and Integration) for generating high-quality text captions for RS images. Firstly, we generate distinct and detailed descriptions from different perspectives using Rule-MLLM(Multimodal Large Language Model) Relay Generation and MLLMs generation methods. Next, we utilize Large Language Models (LLMs) to integrate these diverse descriptions into comprehensive captions, capturing details from multiple perspectives. Finally, we have created the HQRS-IT-210K dataset, including about 210,000 RS images and 1.3 million captions. We fine-tuned two VLFMs using our dataset: CLIP, a discriminative model, and CoCa, an image-to-text generative model. This process resulted in our proposed HQRS-CLIP and RS-CoCa models. Experimental results demonstrate that HQRS-CLIP surpassed the previous SOTA RS CLIP model in various downstream tasks while using only 4.2\% of the training data. RS-CoCa outperforms other advanced approaches across benchmark datasets and can generate captions for RS images that rival or even exceed manual annotations. Dataset, pre-trained models, and codes will be released at https://github.com/YiguoHe/HQRS-210K-and-HQRS-CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
