Zoom-Zero: 通过时间放大实现强化的粗到细视频理解
Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-in
AI总结：
Zoom-Zero通过时间放大和令牌选择性信用分配提升视频问题回答的时空定位精度和答案准确性。
摘要：
基于视频的问题回答（GVQA）旨在定位视频中相关的时序片段并生成准确的答案；然而，大型视频-语言模型（LVLMs）在时间感知方面存在局限。尽管现有基于组相对策略优化（GRPO）的方法试图改进时间定位，但它们仍然难以在相关视频证据中忠实定位其答案，导致时间定位错误和幻觉。在本工作中，我们提出了Zoom-Zero，一个从粗到细的框架，首先定位与查询相关的片段，然后在最显著的帧上进行时间放大以进行更细致的视觉验证。我们的方法通过两个关键创新解决了GRPO在GVQA任务中的局限：（i）一种放大准确性奖励，验证时间定位预测的保真度，并促进对定位帧的细致视觉验证；（ii）令牌选择性信用分配，将奖励分配给负责时间定位或答案生成的令牌，缓解GRPO在处理多方面的奖励信号时的问题。我们提出的方法推进了基于视频的问题回答，提高了NExT-GQA上的时间定位精度5.2%和ReXTime上的时间定位精度4.6%，同时提高了平均答案准确率2.4%。此外，在推理过程中进行的粗到细放大进一步有助于长视频理解，通过在不损害全局上下文的情况下保留关键视觉细节，从而在长视频基准上实现了平均提高6.4%。
英文摘要：
Grounded video question answering (GVQA) aims to localize relevant temporal segments in videos and generate accurate answers to a given question; however, large video-language models (LVLMs) exhibit limited temporal awareness. Although existing approaches based on Group Relative Policy Optimization (GRPO) attempt to improve temporal grounding, they still struggle to faithfully ground their answers in the relevant video evidence, leading to temporal mislocalization and hallucinations. In this work, we present Zoom-Zero, a coarse-to-fine framework that first localizes query-relevant segments and then temporally zooms into the most salient frames for finer-grained visual verification. Our method addresses the limits of GRPO for the GVQA task with two key innovations: (i) a zoom-in accuracy reward that validates the fidelity of temporal grounding prediction and facilitates fine-grained visual verification on grounded frames; (ii) token-selective credit assignment, which attributes rewards to the tokens responsible for temporal localization or answer generation, mitigating GRPO's issue in handling multi-faceted reward signals. Our proposed method advances grounded video question answering, improving temporal grounding by 5.2\% on NExT-GQA and 4.6\% on ReXTime, while also enhancing average answer accuracy by 2.4\%. Additionally, the coarse-to-fine zoom-in during inference further benefits long-form video understanding by preserving critical visual details without compromising global context, yielding an average improvement of 6.4\% on long-video benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
