看见它,说出它,排序:一种无需训练的迭代框架用于LVLMs中的视觉引导多模态推理
See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
AI总结:
本文提出无需训练的迭代框架,通过视觉证据监督推理步骤,减少多模态推理中的幻觉问题,提升推理准确性。
摘要:
最近的大型视觉-语言模型(LVLMs)通过生成长链推理(CoT)响应展示了令人印象深刻的推理能力。然而,在多模态上下文中,CoT推理对视觉幻觉传播高度敏感:一旦中间推理步骤与视觉证据不一致,后续步骤即使逻辑上有效,仍可能导致最终答案错误。现有解决方案试图通过强化学习(RL)训练模型
英文摘要:
Recent large vision-language models (LVLMs) have demonstrated impressive reasoning ability by generating long chain-of-thought (CoT) responses. However, CoT reasoning in multimodal contexts is highly vulnerable to visual hallucination propagation: once an intermediate reasoning step becomes inconsistent with the visual evidence, subsequent steps-even if logically valid-can still lead to incorrect final answers. Existing solutions attempt to mitigate this issue by training models to "think with images" via reinforcement learning (RL). While effective, these methods are costly, model-specific, and difficult to generalize across architectures. Differently, we present a lightweight method that bypasses RL training and provides an iterative, training-free, plug-and-play framework for visually-grounded multimodal reasoning. Our key idea is to supervise each reasoning step at test time with visual evidence, ensuring that every decoded token is justified by corresponding visual cues. Concretely, we construct a textual visual-evidence pool that guides the model's reasoning generation. When existing evidence is insufficient, a visual decider module dynamically extracts additional relevant evidence from the image based on the ongoing reasoning context, expanding the pool until the model achieves sufficient visual certainty to terminate reasoning and produce the final answer. Extensive experiments on multiple LVLM backbones and benchmarks demonstrate the effectiveness of our approach. Our method achieves 16.5%-29.5% improvements on TreeBench and 13.7% RH-AUC gains on RH-Bench, substantially reducing hallucination rates while improving reasoning accuracy without additional training.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。