RSTeller：利用公开数据和大语言模型的丰富语言语义扩展遥感视觉语言建模
RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
AI总结：
本文提出利用大语言模型从公开OSM数据和GEE图像生成大规模遥感多模态数据集RSTeller（含130余万图像），通过持续预训练提升视觉语言模型性能，降低标注成本并促进遥感研究。
摘要：
丰富且标注良好的多模态数据对于将复杂的视觉遥感场景与人类语言对齐至关重要，这使得能够开发用于各种遥感解译任务的专用视觉语言模型。然而，大规模地为遥感图像标注丰富的语言语义需要遥感领域的专业知识和大量人力，这使得标注成本高昂且往往不可行。在本研究中，我们提出了一种工作流程，利用大语言模型从OpenStreetMap（OSM）的普通数据中，为源自Google Earth Engine（GEE）平台的图像生成具有语义丰富描述的多模态数据集。该方法促进了配对遥感数据的生成，并且可以利用公开数据轻松扩展。在此框架下，我们提出了RSTeller，这是一个包含超过130万张遥感图像的多模态数据集，每张图像配有两个描述性标题。大量实验表明，通过持续预训练，RSTeller提升了多个现有视觉语言模型在遥感场景理解方面的性能。我们的方法显著减少了标注遥感图像所需的人工工作和专业知识，同时使高质量标注数据的获取更加民主化。这一进展促进了视觉语言建模的发展，并鼓励更广泛地参与遥感研究和应用。RSTeller数据集可在https://github.com/SlytherinGe/RSTeller获取。
英文摘要：
Abundant, well-annotated multimodal data in remote sensing are pivotal for aligning complex visual remote sensing (RS) scenes with human language, enabling the development of specialized vision language models across diverse RS interpretation tasks. However, annotating RS images with rich linguistic semantics at scale demands expertise in RS and substantial human labor, making it costly and often impractical. In this study, we propose a workflow that leverages large language models (LLMs) to generate multimodal datasets with semantically rich captions at scale from plain OpenStreetMap (OSM) data for images sourced from the Google Earth Engine (GEE) platform. This approach facilitates the generation of paired remote sensing data and can be readily scaled up using openly available data. Within this framework, we present RSTeller, a multimodal dataset comprising over 1.3 million RS images, each accompanied by two descriptive captions. Extensive experiments demonstrate that RSTeller enhances the performance of multiple existing vision language models for RS scene understanding through continual pre-training. Our methodology significantly reduces the manual effort and expertise needed for annotating remote sensing imagery while democratizing access to high-quality annotated data. This advancement fosters progress in visual language modeling and encourages broader participation in remote sensing research and applications. The RSTeller dataset is available at https://github.com/SlytherinGe/RSTeller.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
