EnsemHalDet: 通过内部状态检测器的集成实现鲁棒的视觉语言模型幻觉检测
EnsemHalDet: Robust VLM Hallucination Detection via Ensemble of Internal State Detectors
AI总结：
本文提出EnsemHalDet，一种通过集成多个内部表示的视觉语言模型幻觉检测框架，以提高多模态幻觉检测的鲁棒性。
摘要：
视觉语言模型（VLMs）在多模态任务中表现出色，但它们仍然容易受到事实错误或与输入图像无关的幻觉影响。最近的研究表明，利用内部表示进行幻觉检测比仅依赖模型输出的方法更高效和准确。然而，现有的基于内部表示的方法通常依赖于单一的表示或检测器，限制了它们捕捉多样化幻觉信号的能力。在本文中，我们提出了EnsemHalDet，一种基于集成的幻觉检测框架，利用VLMs的多种内部表示，包括注意力输出和隐藏状态。EnsemHalDet为每个表示训练独立的检测器，并通过集成学习进行组合。在多个VQA数据集和VLMs上的实验结果表明，EnsemHalDet在AUC方面始终优于先前的方法和单检测器模型。这些结果表明，集成多样化的内部信号显著提高了多模态幻觉检测的鲁棒性。
英文摘要：
Vision-Language Models (VLMs) excel at multimodal tasks, but they remain vulnerable to hallucinations that are factually incorrect or ungrounded in the input image. Recent work suggests that hallucination detection using internal representations is more efficient and accurate than approaches that rely solely on model outputs. However, existing internal-representation-based methods typically rely on a single representation or detector, limiting their ability to capture diverse hallucination signals. In this paper, we propose EnsemHalDet, an ensemble-based hallucination detection framework that leverages multiple internal representations of VLMs, including attention outputs and hidden states. EnsemHalDet trains independent detectors for each representation and combines them through ensemble learning. Experimental results across multiple VQA datasets and VLMs show that EnsemHalDet consistently outperforms prior methods and single-detector models in terms of AUC. These results demonstrate that ensembling diverse internal signals significantly improves robustness in multimodal hallucination detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
