通过屏障调控自适应闭式引导缓解视觉语言模型中的幻觉
Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
AI总结：
提出BRACS框架，通过监测视觉注意力并仅在接地退化时进行闭式修正，无需训练即可有效减少LVLM中的物体幻觉。
摘要：
大型视觉语言模型（LVLMs）经常幻觉出输入图像中不存在的物体，这主要是因为随着解码进行，视觉接地减弱。现有的推理时缓解方法在生成过程中修改logits或隐藏状态，但它们存在三个关键限制：缺乏明确的接地目标，即使在模型已经良好接地时也进行干预，以及使用固定的修正强度，无法适应接地失败的严重程度。我们提出BRACS（屏障调控自适应闭式引导），一种无需训练的引导框架，通过屏障调控自适应闭式引导解决这些问题。BRACS监测模型自身的注意力以衡量视觉接地，并仅在接地恶化时对隐藏状态进行修正。修正更新以闭式解析计算，无需训练辅助网络或重新训练模型。在LLaVA-1.5-7B和Qwen-VL-Chat上的实验表明，BRACS在幻觉基准上持续优于先前方法，将CHAIR$_s$降低9.4个点，将POPE F1提高2.7个点，同时在四个通用多模态基准上匹配或提升性能。BRACS还保持高效，运行速度为贪心解码吞吐量的80%，平均速度比基线快1.3倍。
英文摘要：
Large vision-language models (LVLMs) often hallucinate objects that are not present in the input image, largely because visual grounding weakens as decoding progresses. Existing inference-time mitigation methods modify logits or hidden states throughout generation, but they suffer from three key limitations: they lack an explicit grounding objective, intervene even when the model is already well-grounded, and use fixed correction strengths that do not adapt to the severity of grounding failure. We propose BRACS (Barrier-Regulated Adaptive Closed-form Steering), a training-free steering framework that addresses these issues through barrier-regulated adaptive closed-form steering. BRACS monitors the model's own attention to measure visual grounding and applies corrections to the hidden states only when grounding deteriorates. The corrective update is computed analytically in closed form, requiring no training of auxiliary networks or model retraining. Experiments on LLaVA-1.5-7B and Qwen-VL-Chat show that BRACS consistently outperforms prior methods on hallucination benchmarks, reducing CHAIR$_s$ by 9.4 points and improving POPE F1 by 2.7 points, while matching or improving performance on four general multimodal benchmarks. BRACS also remains efficient, operating at 80% of greedy decoding throughput and achieving 1.3 times higher speed on average than the baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
