HII-DPO: 通过准确的 hallucination-Inducing Counterfactual Images 消除幻觉
HII-DPO: Eliminate Hallucination via Accurate Hallucination-Inducing Counterfactual Images
AI总结：
HII-DPO通过生成准确的幻觉诱导图像，有效缓解了VLMs在语言偏见下的幻觉问题，提升了模型的对齐能力。
摘要：
大型视觉-语言模型（VLMs）在多样化的多模态任务中取得了显著的成功，但仍然容易受到根植于内在语言偏见的幻觉的威胁。尽管最近有了进展，现有的幻觉缓解方法往往忽略了由语言偏见驱动的底层幻觉模式。在本工作中，我们设计了一个新的流程，以准确地合成幻觉诱导图像（HIIs）。利用合成的HIIs，我们揭示了一种一致的场景条件幻觉模式：模型倾向于在视觉证据被移除时提及高度典型的场景对象。为了量化VLMs对这种幻觉模式的易感性，我们建立了Masked-Object-Hallucination（MOH）基准，以严格评估现有最先进的对齐框架。最后，我们利用HIIs构建高质量的偏好数据集，以实现精细的对齐。实验结果表明，我们的方法在有效缓解幻觉的同时，能够保持通用模型能力。具体而言，我们的方法在标准幻觉基准上比当前最先进的方法提高了高达38%。
英文摘要：
Large Vision-Language Models (VLMs) have achieved remarkable success across diverse multimodal tasks but remain vulnerable to hallucinations rooted in inherent language bias. Despite recent progress, existing hallucination mitigation methods often overlook the underlying hallucination patterns driven by language bias. In this work, we design a novel pipeline to accurately synthesize Hallucination-Inducing Images (HIIs). Using synthesized HIIs, we reveal a consistent scene-conditioned hallucination pattern: models tend to mention objects that are highly typical of the scene even when visual evidence is removed. To quantify the susceptibility of VLMs to this hallucination pattern, we establish the Masked-Object-Hallucination (MOH) benchmark to rigorously evaluate existing state-of-the-art alignment frameworks. Finally, we leverage HIIs to construct high-quality preference datasets for fine-grained alignment. Experimental results demonstrate that our approach effectively mitigates hallucinations while preserving general model capabilities. Specifically, our method achieves up to a 38% improvement over the current state-of-the-art on standard hallucination benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
