干预所有路径：统一缓解跨对齐格式的大型视觉-语言模型幻觉
Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
AI总结：
本文提出一种统一干预框架，通过分析不同路径间的相互作用，有效缓解跨对齐格式的LVLM幻觉问题。
摘要：
尽管在广泛的任务上表现出色，大型视觉-语言模型（LVLMs）仍容易产生幻觉。在本研究中，我们提出了一种与LVLMs转换器因果架构对齐的全面干预框架，整合了不同干预路径对幻觉的影响。我们发现，LVLMs的幻觉并非来自单一因果路径，而是来自图像到输入文本、图像到输出文本和文本到文本路径之间的相互作用。首次发现，LVLMs根据问题-答案对齐格式依赖于不同的路径。基于这些见解，我们提出简单而有效的办法来识别和干预每个路径中的关键幻觉头，以适应辨别性和生成性格式。在多个基准测试中的实验表明，我们的方法在各种对齐类型中一致地减少了幻觉。
英文摘要：
Despite their impressive performance across a wide range of tasks, Large Vision-Language Models (LVLMs) remain prone to hallucination. In this study, we propose a comprehensive intervention framework aligned with the transformer's causal architecture in LVLMs, integrating the effects of different intervention paths on hallucination. We find that hallucinations in LVLMs do not arise from a single causal path, but rather from the interplay among image-to-input-text, image-to-output-text, and text-to-text pathways. For the first time, we also find that LVLMs rely on different pathways depending on the question-answer alignment format. Building on these insights, we propose simple yet effective methods to identify and intervene on critical hallucination heads within each pathway, tailored to discriminative and generative formats. Experiments across multiple benchmarks demonstrate that our approach consistently reduces hallucinations across diverse alignment types.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
