跨模态注意力校准用于LVLM幻觉缓解
Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
AI总结：
提出一种无需训练的跨模态注意力校准方法，通过设计模态间解码和位置校准模块，缓解大型视觉语言模型中的幻觉问题。
摘要：
大型视觉语言模型（LVLM）在视觉-语言理解方面表现出显著能力。尽管取得了成功，LVLM在复杂生成任务中仍然会产生幻觉，导致视觉输入与生成内容不一致。为了解决这个问题，一些方法引入了推理时干预，如对比解码，以减少对语言先验的过度依赖。然而，这些方法忽略了由位置偏差和虚假跨模态相关性引起的幻觉。在本文中，我们提出了一种跨模态注意力校准（CMAC）方法，以无需训练的方式缓解LVLM中的幻觉。在该方法中，我们设计了一个模态间解码（IMD）模块，通过一种新颖的对比解码机制来减轻幻觉。IMD将具有显著跨模态注意力权重的值向量掩蔽为失真，从而同时解决了单模态过度依赖和误导性跨模态相关性问题。此外，跨模态位置校准（CMPC）模块缩小了图像标记的位置差距，缓解了跨模态注意力中的位置偏差。在多种幻觉基准上的实验结果验证了我们的方法在减少LVLM幻觉方面优于现有最先进技术。我们的代码将在https://github.com/lijm48/IMCCD上提供。
英文摘要：
Large vision-language models (LVLMs) have shown remarkable capabilities in visual-language understanding. Despite their success, LVLMs still suffer from generating hallucinations in complex generation tasks, leading to inconsistencies between visual inputs and generated content. To address this issue, some approaches have introduced inference-time interventions, such as contrastive decoding, to reduce overreliance on language priors. However, these approaches overlook hallucinations stemming from position bias and spurious inter-modality correlations. In this paper, we propose a Cross-Modal Attention Calibration (CMAC) method to mitigate hallucinations in LVLMs in a training-free manner. In this method, we design an Inter-Modality Decoding (IMD) module to alleviate hallucination by a novel contrastive decoding mechanism. IMD masks the value vectors associated with significant cross-modal attention weights as distortion, which addresses both uni-modality overreliance and misleading inter-modality correlations. Additionally, a Cross-Modal Position Calibration (CMPC) module shrinks the position gap of image tokens, alleviating the position bias in cross-modal attention. Experimental results on diverse hallucination benchmarks validate the superiority of our method over existing state-of-the-art techniques in reducing hallucinations for LVLM. Our code will be available at https://github.com/lijm48/IMCCD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
