基于视觉的强化学习用于视觉推理
Grounded Reinforcement Learning for Visual Reasoning
AI总结：
本文提出ViGoRL，通过强化学习实现视觉推理，通过空间坐标锚定推理步骤，提升视觉定位和搜索性能，优于传统方法。
摘要：
尽管强化学习在数学和编码任务中显著提升了语言模型，但视觉推理需要模型引导视觉注意力、解读感知输入并用空间证据支撑抽象推理。我们引入ViGoRL，通过强化学习训练视觉语言模型，将每个推理步骤明确锚定到特定视觉坐标。受人类视觉决策启发，ViGoRL学习生成空间接地的推理轨迹，每一步引导视觉注意力到相关区域。当需要精细探索时，我们的新型多轮强化学习框架使模型能动态放大预测坐标。在多样化的视觉推理基准上，ViGoRL在空间推理、视觉搜索和基于网页的接地任务中均优于监督微调和传统强化学习基线。结合多轮强化学习与放大视觉反馈显著提升了ViGoRL在定位小GUI元素和视觉搜索中的性能，达到86.4%的V*Bench成绩。此外，我们发现接地增强了其他视觉行为，如区域探索、接地子目标设定和视觉验证。最终，人类评估显示模型的视觉参考不仅空间准确，而且有助于理解模型推理步骤。我们的结果表明，视觉接地强化学习是赋予模型通用视觉推理能力的强大范式。
英文摘要：
While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
