arXiv 2405.138652024-05-22cs.CV

ReVideo：通过运动与内容控制重制视频

ReVideo: Remake a Video with Motion and Content Control

Peking University Shenzhen Graduate School-Rabbitpre AIGC Joint Research Laboratory（北京大学深圳研究生院-Rabbitpre AIGC联合研究实验室）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；University of Tokyo（东京大学）；School of Electronic Computer Engineering Peking University（北京大学电子与计算机工程学院）

Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, Jian Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

ReVideo提出一种通过内容与运动控制实现精确局部视频编辑的新方法，采用三阶段训练解耦和时空自适应融合模块，支持内容、运动及两者的独立编辑，并可扩展至多区域编辑。

摘要：

尽管利用扩散模型在视频生成和编辑方面取得了显著进展，但实现精确且局部的视频编辑仍然是一个巨大的挑战。此外，大多数现有的视频编辑方法主要侧重于改变视觉内容，而专门针对运动编辑的研究有限。在本文中，我们提出了一种新颖的尝试——重制视频（ReVideo），它不同于现有方法，允许通过指定内容和运动在特定区域进行精确的视频编辑。内容编辑通过修改第一帧来实现，而基于轨迹的运动控制提供了直观的用户交互体验。ReVideo解决了一个涉及内容与运动控制之间耦合和训练不平衡的新任务。为了解决这个问题，我们开发了一种三阶段训练策略，从粗到细逐步解耦这两个方面。此外，我们提出了一个时空自适应融合模块，以在不同的采样步骤和空间位置整合内容和运动控制。大量实验表明，我们的ReVideo在几个精确视频编辑应用上表现出色，即：（1）在保持运动不变的情况下局部更改视频内容，（2）保持内容不变并定制新的运动轨迹，（3）同时修改内容和运动轨迹。我们的方法还可以无缝地将这些应用扩展到多区域编辑，而无需特定训练，展示了其灵活性和鲁棒性。

英文摘要：

Despite significant advancements in video generation and editing using diffusion models, achieving accurate and localized video editing remains a substantial challenge. Additionally, most existing video editing methods primarily focus on altering visual content, with limited research dedicated to motion editing. In this paper, we present a novel attempt to Remake a Video (ReVideo) which stands out from existing methods by allowing precise video editing in specific areas through the specification of both content and motion. Content editing is facilitated by modifying the first frame, while the trajectory-based motion control offers an intuitive user interaction experience. ReVideo addresses a new task involving the coupling and training imbalance between content and motion control. To tackle this, we develop a three-stage training strategy that progressively decouples these two aspects from coarse to fine. Furthermore, we propose a spatiotemporal adaptive fusion module to integrate content and motion control across various sampling steps and spatial locations. Extensive experiments demonstrate that our ReVideo has promising performance on several accurate video editing applications, i.e., (1) locally changing video content while keeping the motion constant, (2) keeping content unchanged and customizing new motion trajectories, (3) modifying both content and motion trajectories. Our method can also seamlessly extend these applications to multi-area editing without specific training, demonstrating its flexibility and robustness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。