关键位置引导：基于令牌级视觉敏感度引导的LVLMs幻觉缓解
Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation
AI总结：
提出令牌级视觉敏感度引导（TLVS）方法，通过提取令牌级引导向量并自适应调整引导强度，仅在关键解码步骤抑制幻觉，在多个基准上优于现有方法。
摘要：
大型视觉语言模型（LVLMs）取得了快速进展并部署在各种应用中，但幻觉仍然是一个主要挑战。激活引导因其训练开销小和推理时可控制而具有吸引力。然而，我们发现，在自回归解码过程中，视觉条件对令牌预测的影响是稀疏且局部的，许多现有方法对整个序列的图像与非图像差异进行平均，稀释了这些关键信号，导致引导方向信噪比低。此外，许多现有方法应用固定的引导强度，错误分配干预预算，过度扰动非关键令牌，并可能导致不稳定。为了解决这些限制，我们提出了令牌级视觉敏感度引导（TLVS）用于幻觉缓解。我们的方法首先提取令牌级引导向量并进行细化，然后仅在关键位置应用细粒度的、视觉敏感度自适应的引导。这种轻量级、即插即用的机制只需要最少的校准训练，可以应用于各种视觉语言模型。它在每个解码步骤调节引导强度，选择性地抑制易产生幻觉的片段，同时保留基于证据的内容。我们在多个基准上评估TLVS，包括POPE、AMBER、CHAIR（COCO）、MMHal和HallusionBench，证明其相对于先前引导方法的一致改进。
英文摘要：
Large vision language models (LVLMs) have made rapid advancements and are deployed across various applications, yet hallucinations remain a major challenge. Activation steering is appealing due to its minimal training overhead and controllability at inference time. However, we found that during autoregressive decoding, visual conditioning affects token prediction sparsely and locally across decoding steps, and many existing methods that average image-versus-no-image differences over the entire sequence dilute these critical signals, yielding low signal-to-noise ratio steering directions. Additionally, many existing methods apply a fixed steering strength, which misallocates the intervention budget, over-perturbs non-critical tokens, and can cause instability. To address these limitations, we propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation. Our approach first extracts token-level steering vectors and refines them, and then applies fine-grained, visual-sensitivity-adaptive steering only where it matters. This lightweight, plug-and-play mechanism requires only minimal training for calibration and can be applied across diverse vision-language models. It modulates the steering strength at each decoding step, selectively suppressing hallucination-prone spans while preserving evidence-grounded content. We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
