V-ITI: 通过视觉推理时间干预缓解多模态大语言模型中的幻觉
V-ITI: Mitigating Hallucinations in Multimodal Large Language Models via Visual Inference-Time Intervention
AI总结：
V-ITI通过视觉推理时间干预框架，有效缓解多模态大语言模型中的视觉相关幻觉问题，同时保持任务性能。
摘要：
多模态大语言模型（MLLMs）在众多视觉-语言任务中表现出色，但存在幻觉问题，即生成与输入视觉不一致的内容，这在精度敏感领域会降低可靠性。这一问题源于视觉忽视的根本问题，即模型未能充分优先处理输入图像。现有方法通常通过干预注意力分数或输出日志it来缓解幻觉，专注于“如何干预”，但忽略了“何时干预”的前提，导致“过度干预”问题，进而引入新的幻觉和不必要的计算开销。为了解决这一差距，我们首先研究了视觉忽视的机制，并揭示其可以通过MLLMs中的头部级激活模式准确检测。因此，我们提出V-ITI，一个轻量级的视觉推理时间干预框架，集成了一个视觉忽视检测器，通过头部级判别探针识别视觉忽视，以及一个视觉回忆干预器，仅在检测到视觉忽视时通过预存的视觉激活信息调节激活。在八个基准和不同MLLM家族上的广泛实验表明，V-ITI在缓解与视觉相关的幻觉的同时，保持了通用任务性能。
英文摘要：
Multimodal Large Language Models (MLLMs) excel in numerous vision-language tasks yet suffer from hallucinations, producing content inconsistent with input visuals, that undermine reliability in precision-sensitive domains. This issue stems from a fundamental problem of visual neglect, where models fail to adequately prioritize input images. Existing methods typically alleviate hallucinations by intervening in the attention score or output logits, focusing on "how to intervene" but overlooking the prerequisite "when to intervene", which leads to the "over-intervention" problem and subsequently introduces new hallucinations and unnecessary computational overhead. To address this gap, we first investigate the mechanism of visual neglect and reveal it can be accurately detected via head-level activation patterns in MLLMs. We thus propose V-ITI, a lightweight visual inference-time intervention framework integrating a Visual Neglect Detector that identifies visual neglect via head-level discriminative probes and a Visual Recall Intervenor that modulates activations with prestored visual activation information only when the visual neglect is detected. Extensive experiments across eight benchmarks and different MLLM families demonstrate that V-ITI consistently mitigates vision-related hallucinations while preserving general task performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
