以毒攻毒：用于缓解大视觉语言模型中对象幻觉的视觉噪声
Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs
AI总结：
针对大视觉语言模型的对象幻觉问题，提出无需修改基础模型的视觉对抗扰动方法，通过优化视觉噪声增强事实基础，在8个先进模型上验证了缓解幻觉的有效性。
摘要：
大视觉语言模型（LVMs）为大语言模型（LLMs）扩展了视觉感知能力，使其能够处理和解读视觉信息。影响其可靠性的一大挑战是对象幻觉问题，即LVMs可能生成看似合理但与事实不符的信息。我们提出了一种新型视觉对抗扰动（VAP）方法来缓解这一幻觉问题。VAP无需修改基础模型，通过施加经过策略性优化的视觉噪声来减轻LVM的幻觉。该方法将幻觉抑制构建为一个优化问题，利用对抗策略生成有益的视觉扰动，增强模型的事实基础并减少参数化知识偏差。大量实验结果表明，该方法在8个最先进的LVM上持续降低了对象幻觉，在多种评估中验证了其有效性。
英文摘要：
Large vision-language models (LVMs) extend large language models (LLMs) with visual perception capabilities, enabling them to process and interpret visual information. A major challenge compromising their reliability is object hallucination that LVMs may generate plausible but factually inaccurate information. We propose a novel visual adversarial perturbation (VAP) method to mitigate this hallucination issue. VAP alleviates LVM hallucination by applying strategically optimized visual noise without altering the base model. Our approach formulates hallucination suppression as an optimization problem, leveraging adversarial strategies to generate beneficial visual perturbations that enhance the model's factual grounding and reduce parametric knowledge bias. Extensive experimental results demonstrate that our method consistently reduces object hallucinations across 8 state-of-the-art LVMs, validating its efficacy across diverse evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
