arXiv 2507.047022025-07-07cs.CVcs.AI

Tempo-R0：一种通过高效时序感知强化学习实现时序视频定位的视频多模态大语言模型

Tempo-R0: A Video-MLLM for Temporal Video Grounding through Efficient Temporal Sensing Reinforcement Learning

Feng Yue, Zhaoxing Zhang, Junming Jiao, Zhengyu Liang, Shiwen Cao, Feifei Zhang, Rong Shen

AI总结：

本文提出视频多模态大语言模型 Tempo-R0，通过自适应注意力分配、显式时间戳模态对齐以及基于部分无关拒绝的分组相对策略优化，在时序视频定位任务上较 SOTA 方法提升约 3.5%。

摘要：

时序视频定位（Temporal Video Grounding，TVG）要求根据语言查询从视频中精准定位相关的时间片段，一直是视频理解领域极具挑战性的任务。与文本或图像相比，视频通常包含更大的信息量和更多的冗余，模型需要对整个视频具备全面的理解，才能准确检索出与查询相关的片段。为此，我们提出 Tempo-R0：一种视频多模态大语言模型（Video-MLLM），通过多模态时序感知强化来实现时序视频定位任务。具体而言，在流程的预处理阶段，我们采用基于帧内容变化的自适应注意力分配（Self-adaptive Attention Allocation，SAA）方法，以高效利用多模态大语言模型有限的注意力。同时，我们还使用显式时间戳模态对齐（Explicit Timestamp-modal Aligned，ETA）方法来增强模型感知视频中事件边界的能力。在流程的微调阶段，我们创造性地将基于部分无关拒绝的分组相对策略优化（Partial Irrelevance Refusing-based Group Relative Policy Optimization，PIR-GRPO）应用于时序视频定位领域，以促进模型不仅接受相关的视频-查询对，而且拒绝无关的视频-查询对，从而提升时序推理能力。实验表明，在原始 QVHighlights 测试基准及其具有更合理真值标注的修正版本上，我们的方法均比 SOTA 方案取得了约 3.5% 的显著优势。

英文摘要：

Temporal Video Grounding (TVG), which requires pinpointing relevant temporal segments from video based on language query, has always been a highly challenging task in the field of video understanding. Videos often have a larger volume of information and redundancy than texts or images. Models should present comprehensive understanding of the whole video to accurately retrieve query-relevant clips. We thus propose Tempo-R0: a Video Multimodal Large Language Model (Video-MLLM) for the temporal video grounding task via multimodal temporal sensing reinforcement. Specifically, during the preprocessing stage of our pipeline, we employ Self-adaptive Attention Allocation (SAA) method based on frame content variation to efficiently use the MLLM's limited attention. The Explicit Timestamp-modal Aligned (ETA) method is also utilized to strengthen our model's capability to perceive the boundaries of events in the video. In the fine-tuning part of our pipeline, we creatively apply Partial Irrelevance Refusing-based Group Relative Policy Optimization (PIR-GRPO) in TVG area to foster model's temporal reasoning from not only accepting relevant video-query pairs but also refusing irrelevant ones. Experiments demonstrate that our method accomplishes a notable advantage over SOTA solutions by around 3.5% on both the original QVHighlights testbench and its corrected version with more reasonable ground truth annotations.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。