Video-STR：利用关系图强化多模态大语言模型在视频时空推理中的能力
Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph
AI总结：
提出Video-STR方法，利用基于图的GRPO推理机制和STV-27k数据集强化MLLMs的视频时空推理能力，在STI-Bench上超越基础模型13%。
摘要：
多模态大语言模型（MLLMs）的最新进展展示了强大的语义理解能力，但在执行精确的时空理解方面存在困难。现有的时空方法主要关注视频本身，而忽略了视频内部的物理信息，如多对象布局和运动。这些限制阻碍了MLLMs在对高精度有要求的下游应用中的使用，包括具身智能和VR。为解决这一问题，我们提出了Video-STR，一种基于图的强化方法，用于精确的视频时空推理。基于可验证奖励的强化学习（RLVR）提供模型能力的能力，我们引入了一种使用基于图的群体相对策略优化（GRPO）方法的推理机制，以帮助模型在思考过程中推断问题潜在的时空拓扑。为解决时空训练数据缺乏的问题，我们构建了STV-27k数据集，包含205k个问答对，涵盖室内和室外环境中的动态多对象场景，以支持模型训练。实验表明，Video-STR在各种基准上取得了最先进的结果，在STI-Bench上比基础模型高出13%，证明了我们方法和数据集的有效性。代码、模型和数据将发布。
英文摘要：
Recent progress in Multimodal Large Language Models (MLLMs) has demonstrated strong semantic understanding capabilities, but struggles to perform precise spatio-temporal understanding. Existing spatio-temporal methods primarily focus on the video itself, while overlooking the physical information within the video, such as multi-object layouts and motion. Such limitations restrict the use of MLLMs in downstream applications that demand high precision, including embodied intelligence and VR. To address this issue, we present Video-STR, a novel graph-based reinforcement method for precise Video Spatio-Temporal Reasoning. Building upon the capacity of Reinforcement Learning with Verifiable Reward (RLVR) to improve model abilities, we introduce a reasoning mechanism using graph-based Group Relative Policy Optimization (GRPO) method to guide the model in inferring the underlying spatio-temporal topology of scenarios during the thinking process. To resolve the lack of spatio-temporal training data, we construct the STV-205k dataset with 205k question-answering pairs, covering dynamic multi-object scenes in both indoor and outdoor environments, to support the model training. Experiments show that Video-STR achieves state-of-the-art results on various benchmarks, outperforming the base model by 13% on STI-Bench, and demonstrating the effectiveness of our approach and dataset. Code, model, and data will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
