arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.03347cs.CVcs.AI
收藏

DIVE: 面向主体驱动视频编辑的DINO优化方法

DIVE: Taming DINO for Subject-Driven Video Editing

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);vivo AI Lab(vivo AI实验室);NVIDIA;Adobe Research(Adobe研究院);Shenzhen University(深圳大学);Southeast University(东南大学);Shenzhen University of Advanced Technology(深圳大学先进技术学院)

Yi Huang, Wei Xiong, He Zhang, Chaoqi Chen, Jianzhuang Liu, Mingfu Yan, Shifeng Chen

AI总结:

针对视频编辑中时间一致性与运动对齐难题,提出DIVE框架,利用预训练DINOv2语义特征引导编辑,通过特征对齐轨迹、结合参考图特征训练LoRA实现主体驱动的高质量视频编辑。

摘要:

在扩散模型于图像生成与编辑领域取得成功的基础上,视频编辑近来受到广泛关注。然而,保持时间一致性与运动对齐仍然是一项挑战。为解决这些问题,本文提出DINO引导的视频编辑框架DIVE,该框架旨在支持源视频中的主体驱动编辑,可基于目标文本提示或具有特定身份的参考图像进行编辑。DIVE的核心在于利用从预训练DINOv2模型提取的强大语义特征作为隐式对应关系,引导编辑过程。具体而言,为确保时间运动一致性,DIVE采用DINO特征与源视频的运动轨迹对齐。为实现精确的主体编辑,DIVE将参考图像的DINO特征融入预训练文本到图像模型中,以学习低秩适配(LoRAs),有效配准目标主体的身份。在多样真实世界视频上开展的大量实验表明,该框架可实现高质量编辑结果,具备鲁棒的运动一致性,凸显了DINO对视频编辑的潜在价值。项目主页:https://dino-video-editing.github.io

英文摘要:

Building on the success of diffusion models in image generation and editing, video editing has recently gained substantial attention. However, maintaining temporal consistency and motion alignment still remains challenging. To address these issues, this paper proposes DINO-guided Video Editing (DIVE), a framework designed to facilitate subject-driven editing in source videos conditioned on either target text prompts or reference images with specific identities. The core of DIVE lies in leveraging the powerful semantic features extracted from a pretrained DINOv2 model as implicit correspondences to guide the editing process. Specifically, to ensure temporal motion consistency, DIVE employs DINO features to align with the motion trajectory of the source video. For precise subject editing, DIVE incorporates the DINO features of reference images into a pretrained text-to-image model to learn Low-Rank Adaptations (LoRAs), effectively registering the target subject's identity. Extensive experiments on diverse real-world videos demonstrate that our framework can achieve high-quality editing results with robust motion consistency, highlighting the potential of DINO to contribute to video editing. Project page: https://dino-video-editing.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑