arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20342cs.CV
收藏

ParaVT: 平衡工具先验悖论以实现代理视频强化学习中的并行工具使用

ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning

MiroMind;NTU(国立台湾大学);HKU(香港大学);HKUST(GZ)(香港科技大学(广州));THU(清华大学);LMMs-Lab(LMMs实验室)

Zuhao Yang, Kaichen Zhang, Sudong Wang, Keming Wu, Zhongyu Yang, Bo Li, Xiaojuan Qi, Shijian Lu, Xingxuan Li, Lidong Bing

AI总结:

本文提出ParaVT,一种用于并行视频工具调用的端到端强化学习框架,通过引入PARA-GRPO机制解决工具先验悖论,提升了长视频理解的性能。

摘要:

通过强化学习(RL)训练大型多模态模型(LMMs)以原生调用视频处理工具(如裁剪)已成为实现长视频理解的有前景途径。然而,现有原生RL方法按顺序调度工具调用(即每回合一个):单个错误的裁剪会传播错误而无法得到同伴纠正,多回合工具调用会破坏上下文,且推理成本与回合数成线性关系。我们引入ParaVT,首个多智能体端到端RL训练框架用于并行视频工具调用,通过单个回合内调度多个时间窗口裁剪以获得更干净的上下文和更好的容错能力。然而,将标准RL应用于ParaVT揭示了一个我们称之为工具先验悖论的障碍:预训练的工具先验能够促进工具探索,但也破坏了冷启动的结构格式并暴露了在温度采样下的跳过工具奖励捷径。一个较弱先验LMM的跨模型对比支持这一观点:格式保持稳定但RL触发零工具调用,表明先验强度是格式崩溃和工具探索的共同驱动因素。我们提出PARA-GRPO(Parseability-Anchored和Ratio-gAted GRPO),它通过两种互补机制增强标准RL:(i)仅在最易崩溃的结构标记位置应用目标格式奖励;(ii)每提示帧预算随机化,创建训练提示,其中调用工具会提供可测量的奖励信号,而跳过工具则不会。在六个长视频理解基准测试中,ParaVT在平均上比Qwen3-VL基线提升了7.9%,而PARA-GRPO将训练时间格式合规性从0.13提升到0.64。随着工具能力在现代LMMs中日益内部化,RL必须与由此产生的先验合作,ParaVT提供了一种通用的代理RL配方。代码、数据和模型权重已公开可用。

英文摘要:

Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools (e.g., cropping) has become a promising route to long-video understanding. However, most existing agentic Video-LMMs dispatch tool calls sequentially (i.e., one per turn). Sequential tool calling can propagate localization errors, accumulate irrelevant context, and require an additional main-model generation for each successive crop. We introduce ParaVT, a multi-agent RL framework for Parallel Video Tool calling, dispatching multiple time-window crops in a single turn, allowing evidence from multiple temporal windows to be examined concurrently. Yet applying standard Group Relative Policy Optimization (GRPO) to ParaVT reveals an obstacle we term the Tool Prior Paradox: inherited tool-use priors appear to support exploration while competing with the output format learned during supervised fine-tuning. Standard RL can also settle on tool skipping. A comparison with a model with weaker tool-use priors shows stable formatting but no tool calls during RL, consistent with a trade-off between format stability and tool exploration. We propose PARA-GRPO (Parseability-Anchored and Ratio-gAted GRPO), which augments standard RL with two complementary mechanisms: (i) a format reward that encourages tag closure and penalizes unclosed reasoning blocks, and (ii) a per-prompt randomization of the overview-frame budget to create a reward advantage for tool use when the overview provides insufficient evidence. Across six long-video understanding benchmarks, ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO raising training-time format reward from 0.13 to a peak of 0.64. These findings highlight the need to account for inherited tool-use priors when designing RL methods for long-video understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑