VisionCoach: 通过视觉感知提示强化视频推理
VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting
AI总结：
VisionCoach通过视觉提示指导强化学习，提升视频推理的时空定位能力，实现无需外部工具的高效推理。
摘要：
视频推理需要模型在帧间定位和跟踪相关证据。尽管强化学习（RL）通过可验证奖励提高准确性，但在推理过程中仍难以实现可靠的时空定位。此外，改进定位通常依赖于大规模训练数据或推理时的感知工具，这会增加标注或计算成本。为解决这一挑战，我们提出了VisionCoach，一种输入自适应的RL框架，通过视觉提示作为训练时的指导来提升时空定位。在RL训练中，视觉提示被选择性地应用于具有挑战性的输入以放大相关证据并抑制干扰项。模型通过自我蒸馏内化这些改进，从而在原始视频上直接实现基于证据的推理，而无需在推理时使用视觉提示。VisionCoach包含两个组件：（1）视觉提示选择器，根据视频和问题预测合适的提示类型；（2）时空推理器，通过RL在视觉提示指导下优化，并使用对象感知奖励进行优化，强制对象身份一致性和多区域边界框重叠。大量实验表明，VisionCoach在可比设置下实现了最先进的性能，涵盖了多样化的视频推理、视频理解和时间定位基准（V-STAR、VideoMME、World-Sense、VideoMMMU、PerceptionTest和Charades-STA），同时保持单一高效的推理路径，无需外部工具。我们的结果表明，训练期间的视觉提示提高了基于证据的视频推理能力，而自我蒸馏使模型能够内化这种能力，无需在推理时使用提示。
英文摘要：
Video reasoning requires models to locate and track question-relevant evidence across frames. While reinforcement learning (RL) with verifiable rewards improves accuracy, it still struggles to achieve reliable spatio-temporal grounding during the reasoning process. Moreover, improving grounding typically relies on scaled training data or inference-time perception tools, which increases annotation cost or computational cost. To address this challenge, we propose VisonCoach, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. During RL training, visual prompts are selectively applied to challenging inputs to amplify question-relevant evidence and suppress distractors. The model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference. VisonCoach consists of two components: (1) Visual Prompt Selector, which predicts appropriate prompt types conditioned on the video and question, and (2) Spatio-Temporal Reasoner, optimized with RL under visual prompt guidance and object-aware grounding rewards that enforce object identity consistency and multi-region bounding-box overlap. Extensive experiments demonstrate that VisonCoach achieves state-of-the-art performance under comparable settings, across diverse video reasoning, video understanding, and temporal grounding benchmarks (V-STAR, VideoMME, World-Sense, VideoMMMU, PerceptionTest, and Charades-STA), while maintaining a single efficient inference pathway without external tools. Our results show that visual prompting during training improves grounded video reasoning, while self-distillation enables the model to internalize this ability without requiring prompts at inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
