SkyScript：用于遥感的大规模语义多样化视觉语言数据集
SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing
AI总结：
针对遥感领域缺乏大规模语义多样化图像-文本数据集的问题，提出SkyScript数据集（260万图像-文本对，29K语义标签），通过持续预训练使VLM在零样本场景分类中平均准确率提升6.2%。
摘要：
遥感影像虽然在帮助实现可持续发展目标和应对气候变化方面具有广泛的应用，但尚未受益于近期多功能、任务无关的视觉语言模型（VLM）的进展。一个关键原因是，用于开发VLM的大规模、语义多样化的图像-文本数据集在遥感影像领域仍然缺失。与自然图像不同，遥感影像及其相关的文本描述无法从公共互联网上大规模高效收集。在这项工作中，我们通过使用地理坐标自动将开放的、未标记的遥感影像与OpenStreetMap中涵盖的丰富语义连接起来，从而弥合了这一差距，并构建了SkyScript，一个针对遥感影像的综合视觉语言数据集，包含260万对图像-文本对，覆盖29K个不同的语义标签。通过在该数据集上进行持续预训练，我们获得了一个VLM，其在七个基准数据集上的零样本场景分类中平均准确率比基线模型高出6.2%。该模型还展示了在细粒度对象属性分类和跨模态检索方面的零样本迁移能力。我们希望该数据集能够支持VLM在遥感中各种多模态任务（如开放词汇分类、检索、图像描述和文本到图像合成）的进步。
英文摘要：
Remote sensing imagery, despite its broad applications in helping achieve Sustainable Development Goals and tackle climate change, has not yet benefited from the recent advancements of versatile, task-agnostic vision language models (VLMs). A key reason is that the large-scale, semantically diverse image-text dataset required for developing VLMs is still absent for remote sensing images. Unlike natural images, remote sensing images and their associated text descriptions cannot be efficiently collected from the public Internet at scale. In this work, we bridge this gap by using geo-coordinates to automatically connect open, unlabeled remote sensing images with rich semantics covered in OpenStreetMap, and thus construct SkyScript, a comprehensive vision-language dataset for remote sensing images, comprising 2.6 million image-text pairs covering 29K distinct semantic tags. With continual pre-training on this dataset, we obtain a VLM that surpasses baseline models with a 6.2% average accuracy gain in zero-shot scene classification across seven benchmark datasets. It also demonstrates the ability of zero-shot transfer for fine-grained object attribute classification and cross-modal retrieval. We hope this dataset can support the advancement of VLMs for various multi-modal tasks in remote sensing, such as open-vocabulary classification, retrieval, captioning, and text-to-image synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
