arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.14510cs.CVcs.AIcs.GR
收藏

V-LASIK:使用合成数据实现视频中眼镜的一致性移除

V-LASIK: Consistent Glasses-Removal from Videos Using Synthetic Data

Lightricks(莱特里克斯公司);Tel Aviv University(特拉维夫大学);Reichman University(赖希曼大学)

Rotem Shalev-Arkushin, Aharon Azulay, Tavi Halperin, Eitan Richardson, Amit H. Bermano, Ohad Fried

AI总结:

针对局部视频编辑中眼镜移除的一致性难题,提出弱监督方法,利用调整后的预训练扩散模型生成合成数据,实现保留内容的一致编辑,泛化至贴纸移除且性能显著优于现有方法。

摘要:

基于扩散的生成模型最近展现出卓越的图像和视频编辑能力。然而,局部视频编辑(尤其是移除眼镜等小属性)仍是一项挑战。现有方法要么过度修改视频、生成不真实的伪影,要么无法在整个视频中一致地执行所需编辑。本研究聚焦于视频中眼镜的一致性、身份保留移除,将其作为视频中一致性局部属性移除的案例研究。由于缺乏配对数据,我们采用弱监督方法,使用调整后的预训练扩散模型生成合成的不完美数据。我们表明,尽管数据不完美,但通过从生成的数据中学习并利用预训练扩散模型的先验知识,我们的模型能够在保留原始视频内容的同时一致地执行所需编辑。此外,我们通过将其成功应用于面部贴纸移除,例证了我们方法对其他局部视频编辑任务的泛化能力。我们的方法相比现有方法展现出显著改进,凸显了利用合成数据和强大视频先验进行局部视频编辑任务的潜力。

英文摘要:

Diffusion-based generative models have recently shown remarkable image and video editing capabilities. However, local video editing, particularly removal of small attributes like glasses, remains a challenge. Existing methods either alter the videos excessively, generate unrealistic artifacts, or fail to perform the requested edit consistently throughout the video. In this work, we focus on consistent and identity-preserving removal of glasses in videos, using it as a case study for consistent local attribute removal in videos. Due to the lack of paired data, we adopt a weakly supervised approach and generate synthetic imperfect data, using an adjusted pretrained diffusion model. We show that despite data imperfection, by learning from our generated data and leveraging the prior of pretrained diffusion models, our model is able to perform the desired edit consistently while preserving the original video content. Furthermore, we exemplify the generalization ability of our method to other local video editing tasks by applying it successfully to facial sticker-removal. Our approach demonstrates significant improvement over existing methods, showcasing the potential of leveraging synthetic data and strong video priors for local video editing tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑