SDCD: 结构破坏对比解码用于缓解大视觉-语言模型中的幻觉
SDCD: Structure-Disrupted Contrastive Decoding for Mitigating Hallucinations in Large Vision-Language Models
AI总结：
SDCD通过结构破坏对比解码缓解大视觉-语言模型中的幻觉问题，有效抑制纹理驱动偏见，提升多模态能力。
摘要：
大型视觉-语言模型（LVLMs）在多模态理解和推理方面取得了显著进展，但物体幻觉仍是一个关键挑战。尽管现有研究侧重于减轻语言先验或高层统计偏见，但它们往往忽视了视觉编码过程的内部复杂性。我们发现，由于视觉编码器在弱结构监督下固有的袋状补丁行为，视觉统计偏见成为物体幻觉的一个促成因素。在这一偏见下，模型倾向于在单个补丁内优先选择局部纹理特征，而非整体几何结构。这种倾向可能导致虚假的视觉信心并导致幻觉。为了解决这一问题，我们引入了一种无需训练的算法，称为结构破坏对比解码（SDCD），通过引入一种打乱结构的视图来进行输出分布的对比校准。通过惩罚在这一无结构视图下保持高置信度的标记，SDCD有效抑制了由纹理驱动的偏见。实验结果表明，SDCD在多个基准测试中显著减轻了幻觉，并增强了LVLMs的整体多模态能力。
英文摘要：
Large Vision-Language Models (LVLMs) demonstrate significant progress in multimodal understanding and reasoning, yet object hallucination remains a critical challenge. While existing research focuses on mitigating language priors or high-level statistical biases, they often overlook the internal complexities of the visual encoding process. We identify that visual statistical bias, arising from the inherent Bag-of-Patches behavior of Vision Encoders under weak structural supervision, acts as a contributing factor of object hallucinations. Under this bias, models prioritize local texture features within individual patches over holistic geometric structures. This tendency may induce spurious visual confidence and result in hallucinations. To address this, we introduce a training-free algorithm called Structure-Disrupted Contrastive Decoding (SDCD), which performs contrastive calibration of the output distribution by introducing a shuffled structure-disrupted view. By penalizing tokens that maintain high confidence under this structure-less view, SDCD effectively suppresses the texture-driven bias. Experimental results demonstrate that SDCD significantly mitigates hallucinations across multiple benchmarks and enhances the overall multimodal capabilities of LVLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
