arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.04415cs.CL
收藏

STEER: 通过多目标强化学习进行视频推理的结构事件证据

When Rewards Disagree: Reward Weighting for Token-Efficient Video Reasoning

University of Western Australia(西澳大学);CUHKSZ(香港科技大学);Tencent Youtu Lab(腾讯优图实验室);SUSTech(南方科技大学);Monash University(墨尔本大学)

Zinuo Li, Yongxin Guo, Jun Liu, Jiawei Zhan, Xi Jiang, Mohammed Bennamoun, Farid Boussaid, Feng Zheng, Qiuhong Ke

AI总结:

本文提出结构事件证据方法,通过多目标强化学习解决视频推理中事件结构缺失问题,提出STEER-60K数据集和P-FAB算法提升推理效率与准确性。

摘要:

人类对视频动态的理解依赖于在进行抽象推理前形成实体、动作和时间关系的结构化表示。相比之下,现有视频大语言模型直接对原始视觉标记应用无结构的推理链,其中关键时间线索被埋没在冗长叙述中,事件级结构被 largely 忽略。我们提出了结构事件证据,将视频表示为紧凑的时间有序事件模式,捕捉显著事件及其关键属性和事件间的时间依赖关系,通过受约束的验证过程实现证据导向的推理。此设计促进简洁、可解释的推理,同时减少无约束推理链中常见的漂移。为在该范式下训练模型,我们引入STEER-60K数据集,包含四个阶段的渐进管道:证据训练、格式预热、推理预热和RL后训练。在RL过程中,推理链长度和任务准确性常冲突,而对困难样本的奖励过于稀疏,导致策略忽视具有挑战性的实例。我们将此问题建模为多目标帕累托最优问题,并提出帕累托前沿引导的优势平衡(P-FAB),动态解决奖励冲突并识别帕累托前沿上的平衡优化方向。最终模型STEER-4B在视频理解任务中与7B规模基线模型表现相当,仅使用一半的输入帧。代码和数据将被发布。

英文摘要:

Human perception interprets video dynamics by constructing structured mental representations of entities, actions, and temporal boundaries before engaging in high-level reasoning. In contrast, current Video-LLMs typically rely on unstructured Chain-of-Thought (CoT), where crucial visual cues are obscured by verbose narration. While recent efforts ground reasoning in intermediate visual evidence, they rely on task-specific schemas that fail to generalize across diverse video understanding tasks. To address this, we introduce Structured Event Evidence, a unified, temporally ordered schema designed for general video reasoning. However, post-training a model to extract structured evidence while reasoning introduces a severe multi-objective optimization bottleneck, as candidate rollouts must simultaneously satisfy schema compliance, task accuracy, and reasoning budget. Standard reinforcement learning (RL) algorithms attempt to balance these objectives via static scalarization, enforcing a rigid accuracy-length trade-off across training and obscuring fine-grained reward signals. We resolve this by proposing Group-Relative Advantage Balancing (GRAB), which dynamically computes Pareto-optimal objective weights within each rollout group via a minimum-norm criterion over centered rewards. Operating strictly in reward space, GRAB incurs virtually zero computational overhead (<0.002% relative runtime to GRPO) and eliminates brittle weight tuning. Supported by our new PARSE-60K dataset (60K annotations across 35K videos) and a three-stage progressive curriculum, we train PARSE-4B. Across temporal grounding and video question-answering benchmarks, GRAB generates 13% to 39% fewer tokens while maintaining competitive accuracy with exhaustively hand-tuned static weights -- without requiring hyperparameter weight sweeps.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑