遥感图像描述中多语言视觉-语言学习的基准
A Benchmark for Multi-Lingual Vision-Language Learning in Remote Sensing Image Captioning
AI总结：
针对遥感图像描述缺乏非英语数据集和多语言评估的问题，本文构建了双语数据集BRSIC并提出系统评估框架，对八种大型视觉-语言模型在多语言任务中的表现进行了全面实证研究。
摘要：
遥感图像描述（RSIC）是一个连接视觉与语言的跨模态领域，旨在自动生成遥感图像中特征和场景的自然语言描述。尽管在为训练视觉-语言模型（VLMs）开发复杂方法和大规模数据集方面取得了重大进展，但两个关键挑战依然存在：非英语描述数据集的稀缺以及模型多语言能力评估的缺乏。这些限制从根本上阻碍了RSIC的进步和实际部署，特别是在大型VLMs时代。为了应对这些挑战，本文为该领域做出了几项重要贡献。首先，我们引入并分析了BRSIC（Bilingual Remote Sensing Image Captioning），这是一个全面的双语数据集，通过中文描述丰富了三个已有的英文RSIC数据集，包含13,634张图像和68,170个双语字幕。在此基础之上，我们开发了一个系统评估框架，解决了评估协议中普遍存在的不一致问题，通过在BRSIC上的标准化重训练程序实现对模型性能的严格评估。此外，我们对八种最先进的大型视觉-语言模型（LVLMs）进行了广泛的实证研究，考察了它们在多种范式下的能力，包括零样本推理、监督微调和多语言训练。这一全面评估为当前LVLMs在处理多语言遥感任务时的优势和局限性提供了关键见解。另外，我们的跨数据集迁移实验也揭示了有趣的发现。代码和数据将发布于https://github.com/mrazhou/BRSIC。
英文摘要：
Remote Sensing Image Captioning (RSIC) is a cross-modal field bridging vision and language, aimed at automatically generating natural language descriptions of features and scenes in remote sensing imagery. Despite significant advances in developing sophisticated methods and large-scale datasets for training vision-language models (VLMs), two critical challenges persist: the scarcity of non-English descriptive datasets and the lack of multilingual capability evaluation for models. These limitations fundamentally impede the progress and practical deployment of RSIC, particularly in the era of large VLMs. To address these challenges, this paper presents several significant contributions to the field. First, we introduce and analyze BRSIC (Bilingual Remote Sensing Image Captioning), a comprehensive bilingual dataset that enriches three established English RSIC datasets with Chinese descriptions, encompassing 13,634 images paired with 68,170 bilingual captions. Building upon this foundation, we develop a systematic evaluation framework that addresses the prevalent inconsistency in evaluation protocols, enabling rigorous assessment of model performance through standardized retraining procedures on BRSIC. Furthermore, we present an extensive empirical study of eight state-of-the-art large vision-language models (LVLMs), examining their capabilities across multiple paradigms including zero-shot inference, supervised fine-tuning, and multi-lingual training. This comprehensive evaluation provides crucial insights into the strengths and limitations of current LVLMs in handling multilingual remote sensing tasks. Additionally, our cross-dataset transfer experiments reveal interesting findings. The code and data will be available at https://github.com/mrazhou/BRSIC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
