arXiv 2511.205152026-03-24cs.CV

HalDec-Bench：图像描述中幻觉检测的基准测试

HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning

OMRON SINICX；The University of Tokyo（东京大学）；The University of Osaka（大阪大学）

Kuniaki Saito, Risa Shinoda, Shohei Tanaka, Tosho Hirasawa, Fumio Okura, Yoshitaka Ushiku

备注：: Previously this version appeared as arXiv:2603.15253 which was submitted as a new work by accident

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出HalDec-Bench，用于评估图像描述中幻觉检测的性能，通过多样化VLM生成的描述和人类标注揭示模型差异，发现检测器倾向于认可响应开头的句子，并通过强VLM过滤减少数据噪声。

摘要：

图像描述中的幻觉检测（HalDec）评估视觉-语言模型对图像内容与文本对齐的能力，通过识别错误描述来检测幻觉。除了评估，有效的幻觉检测对于筛选高质量图像-描述对以训练VLMs至关重要。然而，由于缺乏全面的基准，VLMs在不同描述模型和幻觉类型中的泛化能力仍不清楚。本文引入HalDec-Bench，一个设计用于原则性和可解释性评估幻觉检测器的基准。HalDec-Bench包含由多样化VLM生成的描述，以及指示幻觉存在的标注，详细的幻觉类型类别和段落级标签。该基准提供不同难度的任务，揭示现有多模态推理或对齐基准中不可见的模型性能差异。我们的分析进一步揭示了两个关键发现：第一，检测器倾向于将响应开头的句子视为正确，无论其实际正确性；第二，实验表明，通过使用强VLM作为过滤器，可以显著减少数据噪声。我们的项目页面可在https://dahlian00.github.io/HalDec-Bench-Page/上访问。

英文摘要：

Hallucination detection in captions (HalDec) assesses a vision-language model's ability to correctly align image content with text by identifying errors in captions that misrepresent the image. Beyond evaluation, effective hallucination detection is also essential for curating high-quality image-caption pairs used to train VLMs. However, the generalizability of VLMs as hallucination detectors across different captioning models and hallucination types remains unclear due to the lack of a comprehensive benchmark. In this work, we introduce HalDec-Bench, a benchmark designed to evaluate hallucination detectors in a principled and interpretable manner. HalDec-Bench contains captions generated by diverse VLMs together with human annotations indicating the presence of hallucinations, detailed hallucination-type categories, and segment-level labels. The benchmark provides tasks with a wide range of difficulty levels and reveals performance differences across models that are not visible in existing multimodal reasoning or alignment benchmarks. Our analysis further uncovers two key findings. First, detectors tend to recognize sentences appearing at the beginning of a response as correct, regardless of their actual correctness. Second, our experiments suggest that dataset noise can be substantially reduced by using strong VLMs as filters while employing recent VLMs as caption generators. Our project page is available at https://dahlian00.github.io/HalDec-Bench-Page/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。