ClearSight：多模态大语言模型中对象幻觉缓解的视觉信号增强
ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models
AI总结：
针对多模态大语言模型对象幻觉问题，提出即插即用的视觉放大融合技术，增强中间层视觉信号关注，减少幻觉且不降低推理速度。
摘要：
对比解码策略被广泛用于缓解多模态大语言模型（MLLMs）中的对象幻觉。通过减少对语言先验的过度依赖，这些策略确保生成内容紧密基于视觉输入，产生上下文准确的输出。由于对比解码不需要额外训练或外部工具，它提供了计算效率和通用性，使其极具吸引力。然而，这些方法存在两个主要限制：（1）粗暴地抑制语言先验可能损害生成内容的连贯性和准确性；（2）处理对比输入增加了计算负担，显著降低推理速度。为解决这些挑战，我们提出了视觉放大融合（VAF），一种即插即用技术，增强模型中间层（模态融合主要发生处）对视觉信号的关注。该方法能更有效地捕获视觉特征，减少模型对语言模态的偏见。实验结果表明，VAF在多种MLLMs上显著减少幻觉，且不影响推理速度，同时保持生成输出的连贯性和准确性。
英文摘要：
Contrastive decoding strategies are widely used to mitigate object hallucinations in multimodal large language models (MLLMs). By reducing over-reliance on language priors, these strategies ensure that generated content remains closely grounded in visual inputs, producing contextually accurate outputs. Since contrastive decoding requires no additional training or external tools, it offers both computational efficiency and versatility, making it highly attractive. However, these methods present two main limitations: (1) bluntly suppressing language priors can compromise coherence and accuracy of generated content, and (2) processing contrastive inputs adds computational load, significantly slowing inference speed. To address these challenges, we propose Visual Amplification Fusion (VAF), a plug-and-play technique that enhances attention to visual signals within the model's middle layers, where modality fusion predominantly occurs. This approach enables more effective capture of visual features, reducing the model's bias toward language modality. Experimental results demonstrate that VAF significantly reduces hallucinations across various MLLMs without affecting inference speed, while maintaining coherence and accuracy in generated outputs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
