发表机构
Korea University; Meta AI; KAIST（高丽大学; Meta AI; 韩国科学技术院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对流式视频中查询不可知的在线帧选择问题，提出SVMemAgent，利用GRPO训练动态记忆策略，在在线和离线基准上超越在线基线并媲美离线方法。
AI 中文摘要
大多数关键帧选择研究聚焦于离线设置，假设可以提前获取完整视频和查询。相比之下，现实世界的流式场景要求在视频时长未知的情况下进行在线帧选择，且在选择过程中无法访问查询或未来帧。为解决这一问题，我们引入了流式视频记忆（SVMem），这是一种对先前观察内容的紧凑且具代表性的记忆，随着视频流的展开而持续更新。基于这一设置，我们提出了流式视频记忆智能体（SVMemAgent），它通过在每个时间步决定是用新到达的帧替换现有记忆帧还是丢弃它，来动态维护记忆。SVMemAgent使用组相对策略优化（GRPO）进行训练，奖励来源于多样化的问答对，从而在训练期间隐式地将策略暴露于查询分布，使得在推理时查询不可用的情况下，SVMem能够保留普遍信息丰富的帧。在在线和离线视频基准上的实验表明，SVMemAgent始终优于在线帧选择基线，并与假设可访问完整视频和查询的离线方法达到竞争性性能。通过任务驱动的奖励，SVMemAgent学习到一种涌现的关键帧选择策略，偏好包含文本信息的帧，这可能有利于下游的视频问答任务。
英文摘要
Most keyframe selection studies focus on offline settings, assuming access to the full video and query in advance. In contrast, real-world streaming scenarios require online frame selection under unknown video duration, without access to either the query or future frames during selection. To address this, we introduce Streaming Video Memory (SVMem), a compact and representative memory of previously observed content, updated continuously as the video stream unfolds. Building on this setting, we propose the Streaming Video Memory Agent (SVMemAgent), which dynamically maintains a memory by deciding at each timestep whether to replace an existing memory frame with the incoming frame or discard it. SVMemAgent is trained using Group Relative Policy Optimization (GRPO) with task-driven rewards derived from diverse question-answer pairs, implicitly exposing the policy to a distribution of queries during training so that SVMem retains generally informative frames at inference, when queries are unavailable. Experiments on both online and offline video benchmarks show that SVMemAgent consistently outperforms online frame selection baselines and achieves competitive performance with offline methods that assume access to the full video and query. Through task-driven rewards, SVMemAgent learns an emergent keyframe selection policy that prefers frames containing textual information, which may benefit downstream VideoQA tasks.
