从细粒度视觉差异中学习：通过上下文视觉对比优化缓解多模态幻觉
Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
AI总结：
提出上下文视觉对比优化（IC-VCO）方法，通过共享多图像上下文中的对比图像确保数学严谨的目标，并引入视觉对比蒸馏（VCDist）和对比样本编辑策略，有效缓解多模态幻觉。
摘要：
多模态幻觉仍然是视觉语言模型（VLM）面临的持续挑战。标准的文本直接偏好优化（DPO）由于缺乏显式的视觉监督，往往无法缓解这一问题。虽然现有工作通过将原始图像与负样本对比引入了视觉偏好DPO，但由于配分函数不匹配导致目标在理论上不一致，并且依赖可能引发捷径学习的粗粒度负样本。在这项工作中，我们提出了上下文视觉对比优化（IC-VCO）。通过将对比图像置于共享的多图像上下文中，IC-VCO确保了数学上严谨的目标。我们进一步引入了视觉对比蒸馏（VCDist），一种辅助的可靠性门控正则化器，鼓励多图像对比训练与单图像推理之间的一致性。最后，我们提出了一种对比样本编辑策略，通过精确的语义扰动生成困难负样本。在五个基准上的实验表明，IC-VCO取得了最佳的整体性能，并且我们的样本编辑策略有效。代码和数据可在 https://github.com/OPPO-Mente-Lab/IC-VCO 获取。
英文摘要：
Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
