InsViE-1M：基于精细数据集构建的高效指令式视频编辑
InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction
AI总结：
针对高质量指令式视频编辑数据集匮乏的问题，本文构建了包含100万三元组的InsViE-1M数据集，并提出多阶段学习策略训练InsViE模型，显著提升了视频编辑性能。
摘要：
指令式视频编辑允许仅使用指令进行有效且交互式的视频编辑，而无需额外的输入（如掩码或属性）。然而，收集高质量的训练三元组（源视频、编辑后视频、指令）是一项具有挑战性的任务。现有数据集大多包含低分辨率、短时长且数量有限的源视频，其编辑质量也不尽如人意，这限制了已训练编辑模型的性能。在本研究中，我们提出了一个包含100万个三元组的高质量指令式视频编辑数据集，即InsViE-1M。我们首先精选了高分辨率且高质量的源视频和图像，然后设计了一个有效的编辑-过滤流水线，以构建用于模型训练的高质量编辑三元组。对于源视频，我们使用不同强度的无分类器引导生成其首帧的多个编辑样本，并由GPT-4o根据精心制定的准则进行自动过滤。编辑后的首帧被传播到后续帧以生成编辑后的视频，随后进行另一轮针对帧质量和运动评估的过滤。我们还从高质量图像中生成并过滤了多种视频编辑三元组。借助InsViE-1M数据集，我们提出了一种多阶段学习策略来训练我们的InsViE模型，逐步增强其指令遵循和编辑能力。大量实验证明了我们的InsViE-1M数据集及所训练模型相较于最新技术的优势。代码可在InsViE获取。
英文摘要：
Instruction-based video editing allows effective and interactive editing of videos using only instructions without extra inputs such as masks or attributes. However, collecting high-quality training triplets (source video, edited video, instruction) is a challenging task. Existing datasets mostly consist of low-resolution, short duration, and limited amount of source videos with unsatisfactory editing quality, limiting the performance of trained editing models. In this work, we present a high-quality Instruction-based Video Editing dataset with 1M triplets, namely InsViE-1M. We first curate high-resolution and high-quality source videos and images, then design an effective editing-filtering pipeline to construct high-quality editing triplets for model training. For a source video, we generate multiple edited samples of its first frame with different intensities of classifier-free guidance, which are automatically filtered by GPT-4o with carefully crafted guidelines. The edited first frame is propagated to subsequent frames to produce the edited video, followed by another round of filtering for frame quality and motion evaluation. We also generate and filter a variety of video editing triplets from high-quality images. With the InsViE-1M dataset, we propose a multi-stage learning strategy to train our InsViE model, progressively enhancing its instruction following and editing ability. Extensive experiments demonstrate the advantages of our InsViE-1M dataset and the trained model over state-of-the-art works. Codes are available at \href{https://github.com/langmanbusi/InsViE}{InsViE}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
