arXiv 2605.046412026-07-30cs.CV

CAST：通过字幕引导的视觉注意力调控缓解大型视觉语言模型中的物体幻觉

CAST: Mitigating Object Hallucination in Large Vision-Language Models via Caption-Guided Visual Attention Steering

Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））

Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Libo Qin, Ruihan Chen, Lei Huang, Baohang Li, Kui Jiang, Yaowei Wang, Ting Liu, Bing Qin

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对大型视觉语言模型的物体幻觉问题，提出无需训练的即插即用方法CAST，通过字幕引导的视觉注意力调控，在低推理成本下平均降低物体幻觉6.03%，实现最优性能。

摘要：

尽管大型视觉语言模型（LVLMs）在下游任务上表现出卓越性能，但它们频繁生成与视觉信息不符的内容，导致物体幻觉问题。近期相关工作多依赖昂贵的人工标注和训练成本，或采用会显著增加推理时间的解码策略来解决该问题。本研究观察到，与非字幕查询相比，LVLMs在回答字幕查询时对视觉信息的注意力会显著增强。受此现象启发，我们提出了无需训练、即插即用的幻觉缓解方法——字幕引导的视觉注意力调控（CAST），该方法利用与字幕查询对应的注意力激活模式来增强LVLMs的视觉感知能力。具体而言，我们采用探测技术识别对字幕查询高度敏感的注意力头，并为其输出估计优化的调控方向。这种调控强化了LVLMs的细粒度视觉感知能力，从而有效缓解物体幻觉。CAST在五个广泛使用的LVLMs以及包含判别式和生成式任务的五个基准测试中，平均将物体幻觉降低了6.03%，在实现最先进性能的同时，仅增加极少的推理成本并保留其他基础能力。

英文摘要：

Although Large Vision-Language Models (LVLMs) have demonstrated remarkable performance on downstream tasks, they frequently produce contents that deviate from visual information, leading to object hallucination. To tackle this, recent works mostly depend on expensive manual annotations and training cost, or decoding strategies which significantly increase inference time. In this work, we observe that LVLMs' attention to visual information is significantly enhanced when answering caption queries compared to non-caption queries. Inspired by this phenomenon, we propose Caption-guided Visual Attention Steering (CAST), a training-free, plug-and-play hallucination mitigation method that leverages the attention activation pattern corresponding to caption queries to enhance LVLMs' visual perception capability. Specifically, we use probing techniques to identify attention heads that are highly sensitive to caption queries and estimate optimized steering directions for their outputs. This steering strengthens LVLM's fine-grained visual perception capabilities, thereby effectively mitigating object hallucination. CAST reduced object hallucination by an average of 6.03% across five widely used LVLMs and five benchmarks including both discriminative and generative tasks, demonstrating state-of-the-art performance while adding little inference cost and preserving other foundational capabilities.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。