arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06072cs.CV
收藏

CogVideoX:具有专家Transformer的文本到视频扩散模型

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

Tsinghua University(清华大学);Zhipu AI(智谱AI)
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan … 展开作者

Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, Jie Tang

AI总结:

CogVideoX是基于扩散Transformer的大规模文本到视频生成模型,通过3D VAE、专家Transformer和渐进式训练等技术,生成10秒连贯视频,性能领先。

摘要:

我们提出了CogVideoX,一个基于扩散Transformer的大规模文本到视频生成模型,能够生成与文本提示对齐的10秒连续视频,帧率为16 fps,分辨率为768*1360像素。以前的视频生成模型通常运动有限、持续时间短,并且难以根据文本生成具有连贯叙事的视频。我们提出了几种设计来解决这些问题。首先,我们提出了一个3D变分自编码器(VAE),在空间和时间两个维度上压缩视频,以提高压缩率和视频保真度。其次,为了提高文本-视频对齐,我们提出了一个具有专家自适应层归一化(expert adaptive LayerNorm)的专家Transformer,以促进两种模态之间的深度融合。第三,通过采用渐进式训练和多分辨率帧打包技术,CogVideoX擅长生成连贯、长持续时间、不同形状且具有显著运动的视频。此外,我们开发了一个有效的文本-视频数据处理流水线,包括各种数据预处理策略和视频字幕生成方法,极大地促进了生成质量和语义对齐。结果表明,CogVideoX在多个机器指标和人类评估中均表现出最先进的性能。3D因果VAE、视频字幕模型和CogVideoX的模型权重均公开在https://github.com/THUDM/CogVideo。

英文摘要:

We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑