arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.02491cs.CV
收藏

Text2LIVE:文本驱动的分层图像和视频编辑

Text2LIVE: Text-Driven Layered Image and Video Editing

Weizmann Institute of Science(魏茨曼科学研究所);NVIDIA Research(英伟达研究院)

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, Tali Dekel

摘要:

我们提出了一种zero-shot的方法,文本驱动的外观操纵自然图像和视频。给定输入图像或视频和目标文本提示,我们的目标是编辑现有对象的外观(例如,对象的纹理)或用视觉效果增强场景(例如,烟,火)以语义上有意义的方式。我们使用从单个输入(图像或视频和目标文本提示)中提取的训练样本的内部数据集训练生成器,同时利用外部预训练的CLIP模型来建立我们的损失。我们的关键思想不是直接生成编辑后的输出,而是生成一个编辑层(颜色+不透明度),该层在原始输入上合成。这使我们能够约束生成过程,并通过直接应用于编辑层的新颖的文本驱动损失来保持对原始输入的高保真度。我们的方法既不依赖于预先训练的生成器,也不需要用户提供的编辑掩码。我们展示了对各种对象和场景的高分辨率自然图像和视频进行本地化的语义编辑。

英文摘要:

We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with visual effects (e.g., smoke, fire) in a semantically meaningful manner. We train a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. We demonstrate localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑