arXiv 2511.213752025-11-26cs.CV

通过边界框思考：通过强化微调提升空间时间视频定位

Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning

ByteDance Intelligent Creation（字节跳动智能创作）；Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Jiao Tong University（上海交通大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）

Xin Gu, Haoji Zhang, Qihang Fan, Jingxuan Niu, Zhipeng Zhang, Libo Zhang, Guang Chen, Fan Chen, Longyin Wen, Sijie Zhu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

STVG-o1通过强化微调提升空间时间视频定位性能，实现无需架构修改的先进结果。

摘要：

空间时间视频定位（STVG）需要从自然语言描述中在未剪辑的视频中同时进行时间和空间定位。尽管多模态大语言模型（MLLMs）具有强大的语言理解能力，但它们在STVG上表现不佳，原因是训练目标不一致和标准视觉编码器中细粒度区域-词对齐弱。为此，我们提出了STVG-o1，这是第一个框架，使现成的MLLMs能够在不进行任何架构修改的情况下实现最先进的STVG性能。我们的方法引入了一种边界框链式思考机制，在最终预测之前显式地在中间步骤进行空间时间位置的推理。我们进一步设计了一种多维强化奖励函数，包括格式、一致性、时间、空间和思考奖励，通过强化微调提供几何感知的监督。在HCSTVG-v1/v2和VidSTG上评估，STVG-o1在HCSTVG上设定了新的最先进结果，在HCSTVG-v1上比最佳任务特定方法高出7.3%的m_tIoU，在VidSTG上与专用模型匹配，并超越了所有现有的基于MLLM的方法。它还展示了在数据集上的强大开放词汇泛化能力，确立了MLLMs作为精确空间时间定位的可行且强大的骨干。我们的代码和模型将被发布。

英文摘要：

Spatio-temporal video grounding (STVG) requires localizing a target object in untrimmed videos both temporally and spatially from natural language descriptions. Despite their strong language understanding, multimodal large language models (MLLMs) underperform on STVG due to misaligned training objectives and weak fine-grained region-word alignment in standard visual encoders. To address this, we propose STVG-o1, the first framework that enables off-the-shelf MLLMs to achieve state-of-the-art STVG performance without any architectural modifications. Our method introduces a bounding-box chain-of-thought mechanism that explicitly reasons about spatio-temporal locations in an intermediate step before producing the final prediction. We further design a multi-dimensional reinforcement reward function consisting of format, consistency, temporal, spatial, and think rewards, which provides geometry-aware supervision through reinforcement fine-tuning. Evaluated on HCSTVG-v1/v2 and VidSTG, STVG-o1 sets new state-of-the-art results on HCSTVG, outperforming the best task-specific method by 7.3\% m\_tIoU on HCSTVG-v1, matching specialized models on VidSTG, and surpassing all existing MLLM-based approaches by large margins. It also demonstrates strong open-vocabulary generalization across datasets, establishing MLLMs as viable and powerful backbones for precise spatio-temporal grounding. Our code and models will be released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。