arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.14792cs.CVcs.AIcs.LG
收藏

Make-A-Video:无文本视频数据的文本到视频生成

Make-A-Video: Text-to-Video Generation without Text-Video Data

Meta AI;University of Rochester(罗切斯特大学)

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, Yaniv Taigman

摘要:

我们提出了Make-A-Video -一种直接将文本到图像(T2 I)生成的巨大进展转化为文本到视频(T2 V)的方法。我们的直觉很简单:从成对的文本图像数据中了解世界是什么样子以及如何描述它,并从无人监督的视频片段中了解世界如何移动。Make-A-Video具有三个优点:(1)它加速了T2 V模型的训练(它不需要从头开始学习视觉和多模态表示),(2)它不需要配对的文本视频数据,以及(3)生成的视频继承了广阔的(美学,幻想等的多样性)。今天的图像生成模型。我们设计了一种简单而有效的方法来构建T2 I模型与新颖而有效的时空模块。首先,我们分解完整的时间U-网络和注意力张量,并在空间和时间上近似它们。其次,我们设计了一个时空流水线,以产生高分辨率和帧速率的视频与视频解码器,插值模型和两个超分辨率模型,可以使各种应用程序,除了T2 V。在所有方面,空间和时间分辨率,对文本的忠实性和质量,Make-A-Video在文本到视频生成方面设置了新的最先进的技术,这是由定性和定量措施决定的。

英文摘要:

We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑