LIVE: 利用图像篡改先验知识进行基于指令的视频编辑
LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing
AI总结：
本文提出LIVE框架，结合大规模高质量图像编辑数据和视频数据提升编辑能力，通过帧级噪声策略和两阶段训练策略，构建包含60多项挑战性任务的评估基准，实验表明方法达到SOTA。
摘要：
视频编辑旨在根据用户意图修改输入视频。最近，端到端训练方法受到广泛关注，通过视频生成或编辑模型构建配对视频编辑数据。然而，与图像编辑相比，视频数据的高标注成本严重限制了视频编辑数据集的规模、质量和任务多样性。为弥合这一差距，我们提出LIVE，一种联合训练框架，利用大规模高质量图像编辑数据和视频数据增强编辑能力。为缓解静态图像与动态视频之间的域差异，我们引入帧级token噪声策略，将特定帧的潜在表示作为推理token，利用大规模预训练视频生成模型创建合理的时序变换。此外，通过清理公共数据集和构建自动化数据管道，我们采用两阶段训练策略来退火视频编辑能力。进一步，我们构建了一个涵盖60多项挑战性任务的综合评估基准，这些任务在图像编辑中普遍存在但在现有视频数据集中稀缺。广泛比较和消融实验表明，我们的方法实现了SOTA性能。源代码将公开发布。
英文摘要：
Video editing aims to modify input videos according to user intent. Recently, end-to-end training methods have garnered widespread attention, constructing paired video editing data through video generation or editing models. However, compared to image editing, the high annotation costs of video data severely constrain the scale, quality, and task diversity of video editing datasets when relying on video generative models or manual annotation. To bridge this gap, we propose LIVE, a joint training framework that leverages large-scale, high-quality image editing data alongside video datasets to bolster editing capabilities. To mitigate the domain discrepancy between static images and dynamic videos, we introduce a frame-wise token noise strategy, which treats the latents of specific frames as reasoning tokens, leveraging large pretrained video generative models to create plausible temporal transformations. Moreover, through cleaning public datasets and constructing an automated data pipeline, we adopt a two-stage training strategy to anneal video editing capabilities. Furthermore, we curate a comprehensive evaluation benchmark encompassing over 60 challenging tasks that are prevalent in image editing but scarce in existing video datasets. Extensive comparative and ablation experiments demonstrate that our method achieves state-of-the-art performance. The source code will be publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
