遥感视觉-语言数据集生成中幻觉的测量与缓解
Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing
AI总结：
本文提出通过集成地图作为外部数据源来增强遥感视觉-语言数据集，生成详细上下文字幕，并引入fMoW-mm多模态数据集及方法来测量和缓解LLM生成文本中的幻觉，在少样本自动目标识别中表现更优。
摘要：
视觉语言模型在多个领域取得了令人瞩目的成果。然而，由于配对图文数据的稀缺，其在遥感领域的应用仍然受限。为弥合这一差距，合成字幕生成引起了关注，传统上依赖于使用元数据或边界框的基于规则的方法。尽管这些方法提供了一些描述，但往往缺乏捕捉复杂广域场景所需的深度。大型语言模型（LLMs）为生成更具描述性的字幕提供了有前景的替代方案，但可能产生通用输出且易产生幻觉。本文提出一种新方法，通过将地图作为外部数据源集成来增强遥感视觉-语言数据集，从而生成详细且上下文丰富的字幕。此外，我们提出了测量和缓解LLM生成文本中幻觉的方法。我们引入了fMoW-mm，一个融合卫星图像、地图、元数据和文本注释的多模态数据集。我们证明了其在少样本设置下用于自动目标识别的有效性，与其他视觉-语言遥感数据集相比取得了更优的性能。
英文摘要：
Vision language models have achieved impressive results across various fields. However, adoption in remote sensing remains limited, largely due to the scarcity of paired image-text data. To bridge this gap, synthetic caption generation has gained interest, traditionally relying on rule-based methods that use metadata or bounding boxes. While these approaches provide some description, they often lack the depth needed to capture complex wide-area scenes. Large language models (LLMs) offer a promising alternative for generating more descriptive captions, yet they can produce generic outputs and are prone to hallucination. In this paper, we propose a new method to enhance vision-language datasets for remote sensing by integrating maps as external data sources, enabling the generation of detailed, context-rich captions. Additionally, we present methods to measure and mitigate hallucinations in LLM-generated text. We introduce fMoW-mm, a multimodal dataset incorporating satellite imagery, maps, metadata, and text annotations. We demonstrate its effectiveness for automatic target recognition in few-shot settings, achieving superior performance compared to other vision-language remote sensing datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
