arXiv 2602.117302026-02-12cs.CV

STVG-R1: 通过强化学习激励视频中实例级推理与 grounding

STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning

Xidian University（西电大学）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Beijing Institute of Technology（北京理工大学）

Xiaowen Zhang, Zhi Gao, Licheng Jiao, Lingling Li, Qing Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

STVG-R1通过强化学习框架提升视频中实例级推理与 grounding 的准确性，实现显著的性能提升和泛化能力。

摘要：

在视觉语言模型（VLMs）中，文本描述与视觉坐标之间的不一致往往导致幻觉。这一问题在密集预测任务如空间-时间视频 grounding（STVG）中尤为严重。先前的方法通常专注于增强视觉-文本对齐或附加辅助解码器。然而，这些策略不可避免地引入了额外的可训练模块，导致显著的标注成本和计算开销。在本工作中，我们提出了一种新的视觉提示范式，以避免跨模态坐标对齐的困难问题。具体而言，我们通过为每个对象分配一个唯一且时间一致的ID，将每帧坐标预测重新表述为紧凑的实例级识别问题。这些ID被嵌入到视频中作为视觉提示，为VLMs提供明确且可解释的输入。此外，我们引入了STVG-R1，这是首个针对STVG的强化学习框架，它采用任务驱动的奖励来联合优化时间准确性、空间一致性以及结构格式正则化。在六个基准上的广泛实验证明了我们方法的有效性。STVG-R1在HCSTVG-v2基准上以m_IoU指标比基线Qwen2.5-VL-7B高出20.9%的显著优势，建立了新的状态-of-the-art（SOTA）。令人惊讶的是，STVG-R1在多对象引用视频对象分割任务中也表现出强大的零样本泛化能力，实现了MeViS上的SOTA 47.3% J&F。

英文摘要：

In vision-language models (VLMs), misalignment between textual descriptions and visual coordinates often induces hallucinations. This issue becomes particularly severe in dense prediction tasks such as spatial-temporal video grounding (STVG). Prior approaches typically focus on enhancing visual-textual alignment or attaching auxiliary decoders. However, these strategies inevitably introduce additional trainable modules, leading to significant annotation costs and computational overhead. In this work, we propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities. Specifically, we reformulate per-frame coordinate prediction as a compact instance-level identification problem by assigning each object a unique, temporally consistent ID. These IDs are embedded into the video as visual prompts, providing explicit and interpretable inputs to the VLMs. Furthermore, we introduce STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization. Extensive experiments on six benchmarks demonstrate the effectiveness of our approach. STVG-R1 surpasses the baseline Qwen2.5-VL-7B by a remarkable margin of 20.9% on m_IoU on the HCSTVG-v2 benchmark, establishing a new state of the art (SOTA). Surprisingly, STVG-R1 also exhibits strong zero-shot generalization to multi-object referring video object segmentation tasks, achieving a SOTA 47.3% J&F on MeViS.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。