arXiv 2406.127182025-03-14cs.CVcs.AIcs.CL

利用全局与局部注意力组合缓解大型视觉-语言模型中的物体幻觉

Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention

Xi’an Jiaotong University（西安交通大学）；National Engineering Laboratory for Big Data Analytics（国家大数据分析工程实验室）；Nanyang Technological University（南洋理工大学）；Lenovo Research（联想研究院）；University of Massachusetts Boston（马萨诸塞大学波士顿分校）；Zhejiang University of Technology（浙江工业大学）

Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, Shijian Lu

备注：: Accepted by CVPR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对 LVLM 因忽视提示相关局部视觉特征而产生物体幻觉的问题，本文提出无需训练、即插即用的 AGLA，通过融合原始图像全局生成特征与增强图像局部判别特征校准 logits，显著缓解幻觉并适用于多类任务。

摘要：

尽管大型视觉-语言模型（LVLMs）在多种多模态任务中取得了巨大成功，但其生成的文本回复往往与图像中的实际物体不一致，从而出现物体幻觉。我们考察了不同的 LVLM，并指出物体幻觉的一个根本原因在于对判别性图像特征的注意力不足。具体而言，LVLMs 通常主要关注与提示无关的全局特征，而非与提示相关的局部特征，这削弱了它们的视觉定位能力并导致物体幻觉。我们提出全局与局部注意力组合（Assembly of Global and Local Attention，AGLA），这是一种无需训练、即插即用的方法，通过同时组合用于回复生成的全局特征和用于视觉判别的局部特征来缓解幻觉。具体而言，我们引入一种图像-提示匹配方案，从图像中捕获与提示相关的局部特征，从而形成输入图像的增强视图，在该视图中与提示相关的内容被突出，而无关干扰被抑制。因此，可借助校准后的 logit 分布缓解幻觉，该分布来自原始图像的生成性全局特征和增强图像的判别性局部特征。大量实验表明 AGLA 在缓解 LVLM 幻觉方面具有优越性，并证明其在判别式和生成式任务上均具有广泛适用性。我们的代码可在 https://github.com/Lackel/AGLA 获取。

英文摘要：

Despite great success across various multimodal tasks, Large Vision-Language Models (LVLMs) often encounter object hallucinations with generated textual responses being inconsistent with the actual objects in images. We examine different LVLMs and pinpoint that one root cause of object hallucinations lies with deficient attention on discriminative image features. Specifically, LVLMs often predominantly attend to prompt-irrelevant global features instead of prompt-relevant local features, undermining their visual grounding capacity and leading to object hallucinations. We propose Assembly of Global and Local Attention (AGLA), a training-free and plug-and-play approach that mitigates hallucinations by assembling global features for response generation and local features for visual discrimination simultaneously. Specifically, we introduce an image-prompt matching scheme that captures prompt-relevant local features from images, leading to an augmented view of the input image where prompt-relevant content is highlighted while irrelevant distractions are suppressed. Hallucinations can thus be mitigated with a calibrated logit distribution that is from generative global features of the original image and discriminative local features of the augmented image. Extensive experiments show the superiority of AGLA in LVLM hallucination mitigation, demonstrating its wide applicability across both discriminative and generative tasks. Our code is available at https://github.com/Lackel/AGLA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。