arXiv 2411.152602025-07-14cs.CVcs.AI

VIVID-10M：面向多功能交互式视频局部编辑的数据集与基线

VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing

Northwest Polytechnical University（西北工业大学）；Beijing Institute of Technology（北京理工大学）；Kuaishou Technology（快手科技）

Jiahao Hu, Tianxiong Zhong, Xuebo Wang, Boyuan Jiang, Xingye Tian, Fei Yang, Pengfei Wan, Di Zhang

备注：: 10 pages, 10 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个大规模混合图像-视频局部编辑数据集VIVID-10M与基线模型VIVID，通过关键帧引导的交互式编辑机制支持实体增删改，在自动化指标和用户研究中均达到SOTA。

摘要：

近年来，基于扩散的图像编辑模型取得了显著进展。然而，实现高质量视频编辑仍是一项重大挑战。主要障碍之一是缺乏基于真实世界数据的开源大规模视频编辑数据集，因为构建此类数据集既耗时又昂贵。此外，视频数据表示需要大量的token，这大幅增加了视频编辑模型的训练成本。最后，当前视频编辑模型的交互性有限，用户通常难以在一次尝试中有效表达其编辑需求。为应对这些挑战，本文引入了数据集VIVID-10M和基线模型VIVID。VIVID-10M是首个大规模混合图像-视频局部编辑数据集，旨在降低数据构建和模型训练成本，包含9.7M样本，涵盖广泛的视频编辑任务。VIVID是在VIVID-10M上训练的多功能交互式视频局部编辑模型，支持实体添加、修改和删除。其核心是提出了一种关键帧引导的交互式视频编辑机制，允许用户迭代编辑关键帧并传播至其他帧，从而降低达到预期结果的延迟。大量实验评估表明，该方法在视频局部编辑中实现了最先进的性能，在自动化指标和用户研究中均超越基线方法。VIVID-10M数据集已在https://kwaivgi.github.io/VIVID/开源。

英文摘要：

Diffusion-based image editing models have made remarkable progress in recent years. However, achieving high-quality video editing remains a significant challenge. One major hurdle is the absence of open-source, large-scale video editing datasets based on real-world data, as constructing such datasets is both time-consuming and costly. Moreover, video data requires a significantly larger number of tokens for representation, which substantially increases the training costs for video editing models. Lastly, current video editing models offer limited interactivity, often making it difficult for users to express their editing requirements effectively in a single attempt. To address these challenges, this paper introduces a dataset VIVID-10M and a baseline model VIVID. VIVID-10M is the first large-scale hybrid image-video local editing dataset aimed at reducing data construction and model training costs, which comprises 9.7M samples that encompass a wide range of video editing tasks. VIVID is a Versatile and Interactive VIdeo local eDiting model trained on VIVID-10M, which supports entity addition, modification, and deletion. At its core, a keyframe-guided interactive video editing mechanism is proposed, enabling users to iteratively edit keyframes and propagate it to other frames, thereby reducing latency in achieving desired outcomes. Extensive experimental evaluations show that our approach achieves state-of-the-art performance in video local editing, surpassing baseline methods in both automated metrics and user studies. The VIVID-10M dataset are open-sourced at https://kwaivgi.github.io/VIVID/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。