HTDC: 通过犹豫触发的微分校准缓解大视觉-语言模型中的幻觉
HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
AI总结：
本文提出HTDC，一种无需训练的解码框架，通过检测层间犹豫信号，仅在易出错步骤激活校准，有效抑制幻觉并保持任务准确性。
摘要：
大型视觉-语言模型（LVLMs）在多模态性能上表现强劲，但仍受不稳定视觉对齐和过度依赖语言先验导致的幻觉影响。现有无需训练的解码方法通常在每次解码步骤都应用校准，引入不必要的计算并可能破坏稳定预测。本文通过识别层间犹豫，即中间层中token偏好波动的简单信号，提出Hesitation-Triggered Differential Calibration（HTDC）解码框架，该框架保留标准全分支推理，并仅在易出错步骤激活校准。当触发时，HTDC通过对比全分支与两个轻量级探针（视觉-无化探针和语义-无化探针），抑制易产生幻觉的候选，同时避免对稳定步骤的不必要干预。在代表性幻觉基准测试中，HTDC一致减少幻觉，同时保持强任务准确性，实现了有效性与计算开销之间的良好权衡。
英文摘要：
Large vision-language models (LVLMs) achieve strong multimodal performance, but still suffer from hallucinations caused by unstable visual grounding and over-reliance on language priors. Existing training-free decoding methods typically apply calibration at every decoding step, introducing unnecessary computation and potentially disrupting stable predictions. We address this problem by identifying layer-wise hesitation, a simple signal of grounding instability reflected by fluctuations in token preference across intermediate layers. Based on this observation, we propose Hesitation-Triggered Differential Calibration (HTDC), a training-free decoding framework that preserves standard full-branch inference and activates calibration only at hesitation-prone steps. When triggered, HTDC contrasts the full branch with two lightweight probes, a visual-nullification probe and a semantic-nullification probe, to suppress hallucination-prone candidates while avoiding unnecessary intervention on stable steps. Experiments on representative hallucination benchmarks show that HTDC consistently reduces hallucinations while maintaining strong task accuracy, achieving a favorable trade-off between effectiveness and computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
