基于指令和证据的对比双流解码用于 grounded 视觉语言推理
Instruction-Evidence Contrastive Dual-Stream Decoding for Grounded Vision-Language Reasoning
AI总结：
本文提出 IECD$^2$ 方法，通过双流解码平衡语言信息和视觉真实性，提升视觉语言推理任务的准确性和减少幻觉。
摘要：
视觉语言模型（VLMs）在遵循指令和开放性视觉语言推理任务中表现出色，但生成的流畅输出往往缺乏对视觉证据的充分支持。先前研究表明，指令提示会加剧这一问题，尤其是在视觉信号不确定或模糊时。为解决此挑战，我们提出了一种解码框架，通过在生成过程中平衡语言信息和视觉真实性。我们的方法，即指令-证据对比双流解码（IECD$^2$），在每个解码步骤中维护两个并行的token概率分布：一个由指令驱动的流，促进表达性和信息性响应；另一个由证据驱动的流，强制在图像中严格接地。这两个流通过对称KL基于对比门进行自适应融合，抑制受语言先验偏好但缺乏视觉证据支持的token，同时在两者分布一致时保留它们。我们在多个数据集上评估了IECD$^2$，涵盖多种生成视觉语言推理任务，如captioning和视觉问答，包括POPE、MME、VQAv2、AMBER和MSCOCO。与最先进的解码方法相比，IECD$^2$在任务准确性和推理性能上表现出一致的改进，并显著减少了幻觉。
英文摘要：
Vision-Language Models (VLMs) exhibit strong performance in instruction following and open-ended vision-language reasoning, yet they frequently generate fluent outputs that are weakly grounded in visual evidence. Prior works have shown that instruction prompting further worsens this issue by amplifying language priors, especially when the visual signal is uncertain or ambiguous. To address this challenge, we propose a decoding framework that explicitly balances linguistic informativeness and visual faithfulness during generation. Our method, Instruction-Evidence Contrastive Dual-Stream Decoding (IECD$^2$), maintains two parallel probability distribution of tokens at each decoding step: an instruction-driven stream that promotes expressive and informative responses, and an evidence-driven stream that enforces strict grounding in the image. These two streams are adaptively fused using a symmetric KL-based contrastive gate, which suppresses tokens favored by language priors but unsupported by visual evidence, while preserving them when both distributions agree. We evaluate IECD$^2$ on multiple datasets spanning various generative vision-language reasoning tasks such as captioning and visual question answering on multiple datasets such as, POPE, MME, VQAv2, AMBER, and MSCOCO. IECD$^2$ demonstrates consistent improvements in task accuracy and reasoning performance with substantial reduction in hallucination compared to state-of-the-art decoding approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
