回望与前行：用于多图像幻觉缓解的跨图像注意力校准与关注偏好学习
Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigation
AI总结：
本文提出CAPL框架，通过跨图像注意力校准和偏好学习缓解多图像幻觉问题，提升模型对跨图像关联的建模能力，并在多个任务中取得稳定性能提升。
摘要：
尽管大型视觉-语言模型（LVLMs）已展现出显著的能力，但在多图像任务中容易产生幻觉。我们归因于现有注意力机制的局限性和跨图像建模的不足。受此启发，我们提出了一种结构化的幻觉缓解框架，包含跨图像注意力校准与偏好学习（CAPL）。CAPL在架构层面显式增强了图像间的交互，同时在训练过程中强化对真实跨图像证据的依赖，从而提升模型对跨图像关联的感知和建模能力。具体而言，我们（i）引入了可选图像标记交互注意力机制，以建立精细的跨图像实体对齐和信息流动；（ii）设计了一种基于跨图像建模的偏好优化策略，该策略对比了在完全跨图像交互下和图像相互不可见时的推理结果，促使模型将预测基于真实的视觉证据，从而缓解由文本先验驱动的错误推理。实验结果表明，CAPL在多种模型架构上均能稳定提升性能，在多图像幻觉和通用基准方面均取得显著提升。值得注意的是，单图像视觉任务的性能保持稳定或略有提升，表明具有强大的泛化能力。
英文摘要：
Although large vision-language models (LVLMs) have demonstrated remarkable capabilities, they are prone to hallucinations in multi-image tasks. We attribute this issue to limitations in existing attention mechanisms and insufficient cross-image modeling. Inspired by this, we propose a structured hallucination mitigation framework involving Cross-Image Attention calibration and Preference Learning (CAPL). CAPL explicitly enhances inter-image interactions at the architectural level while reinforcing reliance on genuine cross-image evidence during training, thereby improving the model's perception and modeling of cross-image associations. Specifically, we (i) introduce a selectable image token interaction attention mechanism to establish fine-grained cross-image entity alignment and information flow; (ii) design a cross-image modeling-based preference optimization strategy that contrasts reasoning outcomes under full inter-image interaction and those obtained when images are mutually invisible, encouraging the model to ground its predictions in authentic visual evidence and mitigating erroneous inferences driven by textual priors. Experimental results demonstrate that CAPL consistently improves performance across multiple model architectures, achieving stable gains on both multi-image hallucination and general benchmarks. Notably, performance on single-image visual tasks remains stable or slightly improves, indicating strong generalization capability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
