基于数字孪生表征强化学习的文本驱动推理视频编辑
Text-Driven Reasoning Video Editing via Reinforcement Learning on Digital Twin Representations
AI总结：
针对现有文本驱动视频编辑方法无法处理隐式语义查询的问题，提出RIVER模型，通过数字孪生表征解耦推理与生成，结合强化学习训练，在自建及公开基准上取得最优性能。
摘要：
文本驱动视频编辑支持用户仅通过文本查询修改视频内容。现有方法需用户提供包含精确空间位置与时间边界的编辑目标明确描述才能修改内容，当用户试图通过引用语义属性或对象关系的隐式查询构思编辑操作时，这类要求并不实用。本文提出推理视频编辑任务，要求视频编辑模型在执行修改前通过多跳推理解读隐式查询以推断编辑目标，同时推出首个尝试解决该复杂任务的模型RIVER（基于推理的隐式视频编辑器）。RIVER通过保留空间关系、时间轨迹与语义属性的视频内容数字孪生表征，将推理过程与生成过程解耦；随后大语言模型联合处理该表征与隐式查询，执行多跳推理确定修改方案，输出结构化指令引导基于扩散的编辑器完成像素级修改。RIVER采用强化学习训练，奖励信号同时评估推理准确率与生成质量。最后，本文推出专门面向推理视频编辑的基准RVEBenchmark，包含100个视频、519条覆盖三个推理复杂度等级与类别的隐式查询。RIVER在提出的RVEBenchmark上表现最优，同时在另外两个视频编辑基准VegGIE和FiVE上也取得了当前最优性能，超过了六个基线方法。
英文摘要：
Text-driven video editing enables users to modify video content only using text queries. While existing methods can modify video content if explicit descriptions of editing targets with precise spatial locations and temporal boundaries are provided, these requirements become impractical when users attempt to conceptualize edits through implicit queries referencing semantic properties or object relationships. We introduce reasoning video editing, a task where video editing models must interpret implicit queries through multi-hop reasoning to infer editing targets before executing modifications, and a first model attempting to solve this complex task, RIVER (Reasoning-based Implicit Video Editor). RIVER decouples reasoning from generation through digital twin representations of video content that preserve spatial relationships, temporal trajectories, and semantic attributes. A large language model then processes this representation jointly with the implicit query, performing multi-hop reasoning to determine modifications, then outputs structured instructions that guide a diffusion-based editor to execute pixel-level changes. RIVER training uses reinforcement learning with rewards that evaluate reasoning accuracy and generation quality. Finally, we introduce RVEBenchmark, a benchmark of 100 videos with 519 implicit queries spanning three levels and categories of reasoning complexity specifically for reasoning video editing. RIVER demonstrates best performance on the proposed RVEBenchmark and also achieves state-of-the-art performance on two additional video editing benchmarks (VegGIE and FiVE), where it surpasses six baseline methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
