面向遥感领域的多语言视觉-语言预训练
Multilingual Vision-Language Pre-training for the Remote Sensing Domain
AI总结：
针对遥感领域CLIP适配多语言输入研究不足的问题，该研究提出RS-M-CLIP模型，结合标准CLIP目标与局部全局表征对齐自监督方法微调多语言CLIP，用翻译扩充的多语言数据训练，在多项视觉-语言任务上达SOTA。
摘要：
基于对比语言-图像预训练（Contrastive Language-Image Pre-training，CLIP）的方法如今被广泛用于支持涉及遥感数据的视觉-语言任务，例如跨模态检索。CLIP向这一特定领域的适配依赖于采用标准对比目标的模型微调，所使用的数据要么是现有的人工标注图像-字幕数据集，要么是从遥感图像的其他标注（如物体类别）衍生而来的图像-字幕对对应的合成数据。不同预训练机制的使用受到的关注较少，仅有少数例外考虑了多语言输入。本研究提出了一种面向遥感领域的新型视觉-语言模型，探索对多语言CLIP模型进行微调，并测试一种自监督方法的使用——该方法基于对齐单张输入图像的局部和全局表征，结合标准CLIP目标共同训练。模型训练依赖于组装已有的遥感图像与英文字幕配对的数据集，随后使用自动机器翻译将字幕翻译成另外九种语言。我们证明翻译后的数据确实有帮助，例如甚至能提升在英语上的性能。我们得到的模型被命名为遥感多语言CLIP（Remote Sensing Multilingual CLIP，RS-M-CLIP），在多种视觉-语言任务中取得了当前最优结果，包括跨模态和多语言图文检索，以及零样本图像分类。
英文摘要：
Methods based on Contrastive Language-Image Pre-training (CLIP) are nowadays extensively used in support of vision-and-language tasks involving remote sensing data, such as cross-modal retrieval. The adaptation of CLIP to this specific domain has relied on model fine-tuning with the standard contrastive objective, using existing human-labeled image-caption datasets, or using synthetic data corresponding to image-caption pairs derived from other annotations over remote sensing images (e.g., object classes). The use of different pre-training mechanisms has received less attention, and only a few exceptions have considered multilingual inputs. This work proposes a novel vision-and-language model for the remote sensing domain, exploring the fine-tuning of a multilingual CLIP model and testing the use of a self-supervised method based on aligning local and global representations from individual input images, together with the standard CLIP objective. Model training relied on assembling pre-existing datasets of remote sensing images paired with English captions, followed by the use of automated machine translation into nine additional languages. We show that translated data is indeed helpful, e.g. improving performance also on English. Our resulting model, which we named Remote Sensing Multilingual CLIP (RS-M-CLIP), obtains state-of-the-art results in a variety of vision-and-language tasks, including cross-modal and multilingual image-text retrieval, or zero-shot image classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
