SAVAA: 通过逐步自适应视觉注意力放大减轻LVLMs中的幻觉
SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification
AI总结：
提出SAVAA框架，通过视觉接地熵估计幻觉风险并自适应调整视觉注意力放大因子，在多个基准上显著减轻大型视觉语言模型的幻觉。
摘要：
最近一系列无需训练的减轻大型视觉语言模型（LVLMs）幻觉的方法，通过在单次前向传递的自回归生成过程中放大对视觉标记的注意力。我们将这种范式称为视觉注意力放大（VAA）。在本文中，我们识别出现有VAA方法的一个双重失败模式，原因是它们在生成步骤中使用固定的放大因子：在某些步骤可能太弱，无法解决幻觉，而在其他步骤太强，引入新的幻觉。受此发现启发，我们提出逐步自适应视觉注意力放大（SAVAA），一种新的VAA框架，它估计每个生成标记的幻觉风险，并使用估计的风险自适应地放大下一个生成步骤的视觉注意力。具体来说，我们引入视觉接地熵（VGE），一种轻量级的幻觉风险估计器，它用视觉接地增强预测熵，为那些不确定、在图像中接地较弱或两者兼有的标记分配更高的风险。在VGE的指导下，SAVAA使用估计的风险校准下一个生成步骤的VAA因子，对高风险步骤应用更强的放大，对低风险步骤应用更弱的放大。在LLaVA-NeXT-7B、Qwen3-VL-8B和InternVL3.5-8B上，SAVAA在生成幻觉基准（如CHAIR、SHR和AMBER）上显著优于基线方法。代码可在https://github.com/JiachengZ01/SAVVA获取。
英文摘要：
A line of recent training-free methods for mitigating hallucinations in large vision-language models (LVLMs) operates by amplifying attention to visual tokens during autoregressive generation within a single forward pass. We refer to this paradigm as visual attention amplification (VAA). In this paper, we identify a dual failure pattern in existing VAA methods caused by their use of a fixed amplification factor across generation steps: it can be too weak at some steps, leaving hallucinations unresolved, while too strong at others, introducing new hallucinations. Motivated by this finding, we propose Step-wise Adaptive Visual Attention Amplification (SAVAA), a new VAA framework that estimates hallucination risk for each generated token and uses the estimated risk to adaptively amplify visual attention at the next generation step. Specifically, we introduce Visual Grounding Entropy (VGE), a lightweight hallucination-risk estimator that augments predictive entropy with visual grounding, assigning higher risk to tokens that are uncertain, weakly grounded in the image, or both. Guided by VGE, SAVAA uses the estimated risk to calibrate the VAA factor for the next generation step, applying stronger amplification to higher-risk steps and weaker amplification to lower-risk steps. Across LLaVA-NeXT-7B, Qwen3-VL-8B, and InternVL3.5-8B, SAVAA significantly outperforms baseline methods on generative hallucination benchmarks such as CHAIR, SHR and AMBER. Code is available at: https://github.com/JiachengZ01/SAVVA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
