SHIELD：通过偏差和脆弱性防御抑制LVLM编码器中的幻觉
SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defense
AI总结：
SHIELD通过减少统计偏差、对抗固有偏差和解决脆弱性，有效抑制LVLM编码器中的对象幻觉。
摘要：
大型视觉-语言模型（LVLMs）在多样化的跨模态任务中表现出色。然而，对象幻觉，即模型生成合理但不准确的对象描述，仍然是一个显著的挑战。与以往专注于LLM组件的研究不同，本文首次将LVLM幻觉追溯到视觉编码器，并识别出三个关键问题：统计偏差、固有偏差和脆弱性。为了解决这些挑战，我们提出了SHIELD，一个无需训练的框架，通过三种策略来减轻幻觉：重新加权视觉令牌以减少统计偏差，引入噪声衍生的令牌以对抗固有偏差，以及应用对抗性攻击与对比解码以解决脆弱性。实验表明，SHIELD在多样化的基准和LVLM家族中有效减轻了对象幻觉。此外，SHIELD在通用LVLM基准上实现了强劲的性能，突显了其广泛的应用性。代码可在https://github.com/hukcc/SHIELD上获得。
英文摘要：
Large Vision-Language Models (LVLMs) excel in diverse cross-modal tasks. However, object hallucination, where models produce plausible but inaccurate object descriptions, remains a significant challenge. In contrast to previous work focusing on LLM components, this paper is the first to trace LVLM hallucinations to visual encoders and identifies three key issues: statistical bias, inherent bias, and vulnerability. To address these challenges, we propose SHIELD, a training-free framework that mitigates hallucinations through three strategies: re-weighting visual tokens to reduce statistical bias, introducing noise-derived tokens to counter inherent bias, and applying adversarial attacks with contrastive decoding to address vulnerability. Experiments demonstrate that SHIELD effectively mitigates object hallucinations across diverse benchmarks and LVLM families. Moreover, SHIELD achieves strong performance on the general LVLM benchmark, highlighting its broad applicability. Code is available at https://github.com/hukcc/SHIELD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
