HalluScope：多模态大语言模型的细粒度幻觉诊断
HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
浏览论文内容
中文总结 AI 辅助
研究针对多模态大语言模型的幻觉问题，提出细粒度幻觉诊断任务，开发数据集并设计奖励函数，训练HalluScope-4B和HalluScope-8B模型，在多个基准测试中达最优，其诊断解释能有效指导目标模型纠正幻觉。
中文摘要 AI 辅助
尽管多模态大语言模型在广泛的视觉语言任务中取得了强大性能，但仍存在幻觉问题，即模型输出与视觉内容、文本上下文或常识知识不一致。现有研究主要通过粗粒度检测解决此问题，然而这些方法提供的诊断信息不足。为填补这一空白，我们提出了针对多模态大语言模型的细粒度幻觉诊断，这是一个联合进行幻觉检测、分类和可解释解释生成的新统一任务。我们开发了自动化数据生成管道并构建了HalluScope-30K数据集，基于此设计了多粒度联合奖励函数并训练了两个诊断模型HalluScope-4B和HalluScope-8B，在多个基准测试中取得了领先性能。此外，诊断驱动的反馈实验表明我们模型产生的细粒度诊断解释能有效指导目标模型纠正幻觉。
英文摘要
Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual content, textual context, or commonsense knowledge. Existing studies primarily address this problem through coarse-grained detection. However, these approaches often provide insufficient diagnostic information for understanding hallucination types and supporting downstream hallucination mitigation. To bridge this gap, we propose fine-grained hallucination diagnosis for MLLMs, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation. We develop an automated data generation pipeline and construct HalluScope-30K, a large-scale diagnostic dataset covering eight sources and five task categories. Based on this dataset, we design a multi-granular joint reward function and train two diagnosis models, HalluScope-4B and HalluScope-8B, which achieve state-of-the-art performance on both the MHALO benchmark and our fine-grained hallucination classification benchmark. Notably, detection and classification are mutually beneficial under joint optimization. Furthermore, diagnosis-driven feedback experiments show that the fine-grained diagnostic explanations produced by our model effectively guide target models to correct their hallucinations, with full diagnosis substantially outperforming all baselines on both Qwen3-VL-8B-Instruct and LLaVA-1.5-7B. Our code, data, and models are available at https://github.com/wkinglin/HalluScope.
发表机构
- Peking University（北京大学）
- Joy Future Academy（京东探索研究院）
机构由 AI 辅助整理，请以论文原文为准。
