arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.13951cs.CV
收藏

文本提示驱动的自回归生成多概念视频定制

Text Prompting for Multi-Concept Video Customization by Autoregressive Generation

University of Maryland(马里兰大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院)

Divya Kothandaraman, Kihyuk Sohn, Ruben Villegas, Paul Voigtlaender, Dinesh Manocha, Mohammad Babaeizadeh

AI总结:

提出一种通过自回归生成和文本提示引导,实现预训练T2V模型的多概念视频定制方法,并利用videoCLIP和DINO分数及人工评估验证效果。

摘要:

我们提出了一种针对预训练文本到视频(T2V)模型的多概念定制方法。直观上,多概念定制视频可以从各个概念的视频流形(非线性)交集中导出,而该交集并不容易找到。我们假设,在文本提示的引导下,顺序且受控地走向视频流形的交集,即可得到解决方案。为此,我们以自回归方式顺序生成各种概念及其相应的交互。我们的方法可以生成包含多个定制概念(主体、动作和背景)的视频,例如泰迪熊跑向棕色茶壶、狗拉小提琴以及泰迪熊在海洋中游泳。我们使用videoCLIP和DINO分数以及人工评估对我们的方法进行了定量评估。本文中结果对应的视频可在 https://github.com/divyakraman/MultiConceptVideo2024 找到。

英文摘要:

We present a method for multi-concept customization of pretrained text-to-video (T2V) models. Intuitively, the multi-concept customized video can be derived from the (non-linear) intersection of the video manifolds of the individual concepts, which is not straightforward to find. We hypothesize that sequential and controlled walking towards the intersection of the video manifolds, directed by text prompting, leads to the solution. To do so, we generate the various concepts and their corresponding interactions, sequentially, in an autoregressive manner. Our method can generate videos of multiple custom concepts (subjects, action and background) such as a teddy bear running towards a brown teapot, a dog playing violin and a teddy bear swimming in the ocean. We quantitatively evaluate our method using videoCLIP and DINO scores, in addition to human evaluation. Videos for results presented in this paper can be found at https://github.com/divyakraman/MultiConceptVideo2024.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑