arXiv 2602.107992026-02-11cs.CVcs.AI

RSHallu: 远程传感多模态大语言模型的双模式幻觉评估与领域定制缓解

RSHallu: Dual-Mode Hallucination Evaluation for Remote-Sensing Multimodal Large Language Models with Domain-Tailored Mitigation

College of Computer Science, Chongqing University（重庆大学计算机科学学院）；Heavy Rainfall Research Center of China（中国强降水研究中心）；CMA Key Open Laboratory of Transforming Climate Resources to Economy, Chongqing Institute of Meteorological Sciences（中国气象局气候资源转化经济重点开放实验室、重庆气象科学研究院）；Chongqing Metropolitan College of Science（重庆科学理工学院）；School of Intelligence Science（智能科学学院）；College of Artificial Intelligence, Harbin Institute of Technology（人工智能学院、哈尔滨工业大学）；BNU-UIC Institute of Artificial Intelligence（北京师范大学-国际学院人工智能与未来网络研究院）；Future Networks, Beijing Normal University at Zhuhai（未来网络、北京师范大学珠海分校）

Zihui Zhou, Yong Feng, Yanying Chen, Guofan Duan, Zhenxi Song, Mingliang Zhou, Weijia Jia

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RSHallu通过双模式检查和领域定制数据集，提升遥感多模态大语言模型的幻觉缓解效果，提高无幻觉率21.63个百分点。

摘要：

多模态大语言模型（MLLMs）在遥感（RS）中的应用日益增加，并在遥感视觉定位（RSVG）、遥感视觉问答（RSVQA）和多模态对话等任务上表现出强大的性能。然而，幻觉（即与输入遥感图像不一致的响应）严重阻碍了其在高风险场景（如应急管理与农业监测）中的部署，并在遥感领域仍处于研究不足的状态。在本工作中，我们提出了RSHallu，一个系统性研究，包含三个成果：（1）我们以遥感为导向构建了幻觉的分类体系，并引入了图像级幻觉以捕捉超出以对象为中心的错误（如模态、分辨率和场景级语义）的遥感特定不一致；（2）我们构建了幻觉基准RSHalluEval（2023个问答对），并实现了双模式检查，通过一个在RSHalluCheck数据集（15396个问答对）上微调的紧凑检查器，支持高精度云审计和低成本可重复的本地检查；（3）我们引入了专门针对领域的数据集RSHalluShield（30000个问答对）用于训练友好的缓解，并进一步提出无需训练的插件策略，包括解码时的logit校正和遥感感知的提示。在具有代表性的RS-MLLMs上，我们的缓解方法在统一协议下将无幻觉率提高了最高21.63个百分点，同时在下游遥感任务（RSVQA/RSVG）上保持了具有竞争力的性能。代码和数据集将被发布。

英文摘要：

Multimodal large language models (MLLMs) are increasingly adopted in remote sensing (RS) and have shown strong performance on tasks such as RS visual grounding (RSVG), RS visual question answering (RSVQA), and multimodal dialogue. However, hallucinations, which are responses inconsistent with the input RS images, severely hinder their deployment in high-stakes scenarios (e.g., emergency management and agricultural monitoring) and remain under-explored in RS. In this work, we present RSHallu, a systematic study with three deliverables: (1) we formalize RS hallucinations with an RS-oriented taxonomy and introduce image-level hallucination to capture RS-specific inconsistencies beyond object-centric errors (e.g., modality, resolution, and scene-level semantics); (2) we build a hallucination benchmark RSHalluEval (2,023 QA pairs) and enable dual-mode checking, supporting high-precision cloud auditing and low-cost reproducible local checking via a compact checker fine-tuned on RSHalluCheck dataset (15,396 QA pairs); and (3) we introduce a domain-tailored dataset RSHalluShield (30k QA pairs) for training-friendly mitigation and further propose training-free plug-and-play strategies, including decoding-time logit correction and RS-aware prompting. Across representative RS-MLLMs, our mitigation improves the hallucination-free rate by up to 21.63 percentage points under a unified protocol, while maintaining competitive performance on downstream RS tasks (RSVQA/RSVG). Code and datasets will be released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。