重新思考视觉忽视：通过上下文偏好引导缓解MLLM幻觉
Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation
AI总结：
针对多模态大语言模型的物体幻觉问题，提出无需训练的上下文偏好激活引导框架（CAS），通过提取上下文偏好向量并在中间早期MLP层注入符号残差来控制信息依赖，有效缓解幻觉且不增加解码延迟。
摘要：
物体幻觉仍然是多模态大语言模型（MLLM）可靠部署的主要障碍。当前的推理时缓解方法主要假设幻觉源于视觉忽视，引导模型增强视觉依赖。相反，我们对多个MLLM的系统干预表明，推动更多视觉依赖可能会加剧某些模型的幻觉，而减少视觉依赖则可能缓解幻觉。这一结果表明，将幻觉单纯归因于视觉不足是不充分的。我们认为，图像作为上下文，同时与模型的参数知识和文本上下文竞争。为此，我们提出了一种无需训练的框架——上下文偏好激活引导（CAS）。它通过两组设计好的冲突样本提取两个语义不同的上下文偏好向量（CPV），并在推理时通过单次符号残差注入到中间早期MLP层来控制信息依赖。实验表明，CAS在不增加解码延迟的情况下显著缓解了物体幻觉，并保持了原生文本生成质量。
英文摘要：
Object hallucination remains a primary obstacle to the reliable deployment of Multimodal Large Language Models (MLLMs). Current inference-time mitigation methods mainly assume hallucinations stem from visual neglect, steering models to enhance visual reliance. In contrast, our systematic interventions on multiple MLLMs show that pushing toward more visual reliance may exacerbate hallucinations on some models, while less may mitigate hallucinations. This result suggests that attributing hallucinations solely to visual insufficiency is underdetermined. We argue that the image, as a context, simultaneously competes with the model's parametric knowledge and the textual context. For this, we propose a training-free framework, Context-Preference Activation Steering (CAS). It extracts two semantically distinct Context Preference Vectors (CPVs) via two small sets of designed conflict samples and applies them via single-pass signed residual injection at mid-early MLP layers during inference to control information reliance. Experiments show that CAS substantially mitigates object hallucinations without increasing decoding latency and preserves native text-generation quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
