arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.06571cs.CVcs.AI
收藏

ModelScope 文本生成视频技术报告

ModelScope Text-to-Video Technical Report

Alibaba Group(阿里巴巴集团)

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, Shiwei Zhang

AI总结:

提出由 Stable Diffusion 演进的 ModelScopeT2V 文本生成视频模型,通过时空块实现帧一致性与平滑运动,并在三项指标上超过现有方法。

摘要:

本文介绍了 ModelScopeT2V,这是一个由文本生成图像合成模型(即 Stable Diffusion)演进而来的文本生成视频合成模型。ModelScopeT2V 融入了时空块,以确保一致的帧生成和平滑的运动过渡。该模型能够在训练和推理期间适应不同的帧数,因此适用于图文和视频文本数据集。ModelScopeT2V 汇集了三个组件(即 VQGAN、文本编码器和去噪 UNet),总计包含 17 亿个参数,其中 5 亿个参数专门用于时序能力。该模型在三项评估指标上均表现出优于最先进方法的性能。代码和在线演示可在 https://modelscope.cn/models/damo/text-to-video-synthesis/summary 获取。

英文摘要:

This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑